Spherical video synopsis generation and visualization framework

Abstract

With the advances in video technology, the advent of spherical video (360° video) recorded using an omnidirectional camera offers a limitless field-of-view (FoV) to the viewers. However, they suffer from the fear of missing out (FOMO) because they can only see a particular FoV at a time. Reviewing a long recorded surveillance video i.e., 24 hours a day is a time-consuming process due to temporal and spatial redundancy. A solution to this problem is to compactly represent the video synopsis by shifting the objects along the time domain. Using a multi-camera setup for surveillance creates blind spots. This problem is solved by using a spherical camera. Therefore, in this paper, we focus on creating and visualizing the video synopsis recorded by the spherical camera. The optimization algorithm plays a key role in condensing the recorded video. Hence, a novel spherical video synopsis optimization framework has been introduced to generate compact videos that eliminate FOMO. The synopsis is generated by shifting objects on the temporal axis and displays them simultaneously by optimizing multiple constraints. It minimizes activity loss, virtual collisions, temporal inconsistencies, and synopsis video length by preserving interactions between objects. The proposed multiobjective optimization includes a new constraint to restrict the number of objects displayed per frame due to the limitation of the human visual system. Direction-based visualization methods have been proposed to improve the viewer’s experience without FOMO. Comparative performance of the proposed framework using the latest metaheuristic optimization algorithms with existing video synopsis optimization algorithms is performed. It is found that chronological disorder ratio and overall virtual collision are minimized effectively through the recent metaheuristics optimization algorithms compared to the related works on video synopsis.

Keywords

Display constraint object-based video synopsis optimization panoramic surveillance video spherical video synopsis

1 Introduction

The spherical camera records the scene as a spherical environment pointing from the center, rather than the rectangular view used in traditional videography [1]. Figure 1 gives the different FoV’s of the input 360° video. With the growing popularity of spherical content, end-users are provided with convenient content consumption [2]. Blind spots are a common problem in multi-camera surveillance setups [3], but spherical camera technology solves this problem by capturing the scene in a realistic and unobstructed manner. To monitor the busy city, the 360° camera must be placed motionless on the spot. At the visualization end, the viewer can see all the sides of the scene (i.e., 360° view), allowing a limitless FoV (i.e., no scenes are overlooked), whereas, with multi-camera surveillance, the viewer has to use the camera switch manually to see the whole scene.

Fig. 1

Different FoV’s of the input 360° video

The steps involved in 360° video processing are recording, stitching, projecting, and visualizing the scene. The internal structure of the omnidirectional camera encompasses multiple cameras placed on a specific rig. It records the entire panoramic scene covering all the events around the camera, excluding the camera itself. Once the scene is recorded, it undergoes stitching using specific stitching software or by the camera themselves to synchronize the events together [4]. After the recording and stitching, the 360° scene will experience a projection from one form to another form based on the viewer’s interest. Generally, they can be projected in three ways: panoramic, cube map, and spherical projection. In this work, equirectangular projection is used. The 360° video is immersive and can be visualized on computers, smartphones, or by Head Mounted Display(HMD). On a computer, using the mouse cursor, different FoV can be visualized by clicking and dragging across the scene. For smartphones and HMD, FoV can be visualized based on their orientation.

With the exponential growth in the number of surveillance videos, searching and retrieving content can be time-consuming. Video abstraction is a common solution to this problem by creating a concise representation of the original video. However, the object-based synopsis of the 360° video is still an unexplored area. It provides a way of creating a condensed video from the input 360° surveillance video, highlighting only the activities of the key object.

Previous works on spherical video summarization ([5–9]), have focused on a fixed viewing angle of at least 12 normal FoV (NFoV) glimpses and the saliency of spherical video scenes is not taken into account. Also, the presence of unlimited FoV leads to FOMO. This motivated us to suggest a spherical video synopsis approaches that rely on visualizing prominent areas with only 4 NFoV glimpses without FOMO.

The human visual system is restricted to handling a fixed number of objects less than eight at the same time [10]. During the spherical surveillance video synopsis investigation, the viewer’s experience of understanding the scene is challenging when they display limitless objects per frame [11]. Therefore, the number of objects to be displayed must be considered. The existing traditional video synopsis works have ignored this concept. This motivated us to propose a multiobjective function to restrict the number of visible objects to the viewers simultaneously.

Our contributions and focus of this paper are in the modules of energy minimization problem formulation. Hence, state-of-the-art methods are employed for the pre-processing and post-processing steps of spherical video synopsis generation like track extraction and tube stitching. Toward this end, this work proposes a novel spherical surveillance framework for video synopsis generation. Also, a direction-based visualization framework is proposed as a classical video visualization approach is inappropriate for spherical videos.

The main contributions of this work are given as follows:

A novel object-based spherical video synopsis generation framework with interaction preservation for 360° surveillance video is proposed.

A novel direction based visualization approach is introduced with only four normal FoV glimpses.

For easy understanding of the scene, a viewer determined number of objects in each spherical frame of the synopsis video is suggested.

A comprehensive performance analysis of various latest optimization algorithms with the existing video synopsis works for both varying and fixed synopsis length is performed.

Exhaustive analysis on the number of activity preserved for the viewer specified duration of synopsis video is presented.

This paper is organized as follows. Section 2 reviews the related research work on traditional video synopsis and 360° video summarization. Section 3 details the proposed method. Moreover, this section describes the spherical synopsis videos’ problem formulation, generation, and visualization. Section 4 presents the proposed work’s experimental results and performance analysis. Finally, the conclusions is expressed in Section 5.

2 Related works

This section gives a brief survey of related works on traditional video synopsis and 360° video summarization.

2.1 Traditional video synopsis

Acha et al. proposed two methods for video synopsis, which are low-level graph optimization and object-based approach [12]. Low-level graph optimization processes every pixel in the video. This method allows pixels to be from any time, which is a difficult problem. Hence an object-based approach is introduced. Pritch et al. suggested a video synopsis method for an infinite video stream [13]. The proposed approach has two phases, an online phase, and a response phase. Pritch et al. suggests two ways to create a condensed synopsis video [14]. It uses a greedy approach to generate a synopsis video from the input. Pritch et al. introduced a clustering-based video synopsis method to search and browse surveillance video [15]. Objects with similar motions are clustered together and displayed simultaneously. Tian et al. proposed an energy minimization-based trajectory mapping method using a genetic algorithm [16]. A synopsis video is generated by classifying the object tracks into four groups. The space-time-based event rearrangement optimization algorithm is suggested by Yi et al. [17]. It outlines the problem as an iterative judgment of event association. Spatial integrity is maintained by introducing an event subsection modification approach. Ruan et al. introduced an online-based dynamic tube rearrangement [18]. It includes a graph coloring approach in which the relationships between object tubes are modeled using dynamic graphs. The approach to generate a synopsis of the outdoor environment is given by Ahmed et al. [19]. Objects are categorized using deep learning mechanisms and grouped into entry and exit areas. Based on the viewer’s request, the energy minimization process generates the synopsis. The hybrid version of simulated annealing and teacher learning-based optimization algorithm is suggested by Ghatak et al. [20]. It provides the best global solution with less computational time. It effectively solves the problem of optimal tube rearrangement. Hybridization of simulated annealing and grey wolf optimizer for generating video synopsis was proposed by Ghatak et al. [21]. The overall efficiency of the synopsis framework is improved by using a Generative Adversarial Network-based foreground extraction approach, namely, multi-frame and multiscale in Generative Adversarial Network. The survey of works related to traditional video synopsis is given in Table 1. Existing video synopsis methods for traditional video are investigated by [22–24]. Through the literature study, it is known that there are numerous works available for the generation of synopsis videos from traditional videos.

Table 1
Survey on Traditional Video Synopsis

Authors Motive Optimized parameters Classification

Acha et al. Low level Video Synopsis Generation Activity, occlusion, and length cost ✗

Pritch et al. [13] Video Synopsis Activity, collision, and temporal consistency cost ✗

Pritch et al. [14] Video Synopsis and Video Indexing Activity and discontinuity cost ✗

Pritch et al. [15] Non-chrono-logical Video Synopsis Motion and collision cost ✓

Tian et al. [16] Video Synopsis Activity, the cost-related positive sequence between objects, false collision and real collision cost ✓

Yi et al. [17] Video Synopsis Temporal model, density model and collision model cost ✗

Ruan et al. [18] Online Video Synopsis Tube collision, activity completeness and interaction preservation cost ✗

Ahmed et al. [19] Intelligent Traffic Monitoring Activity, collision and temporal consistency cost ✓

Ghatak et al. [20] Video Synopsis Activity, collision, and temporal consistency cost ✗

Ghatak et al. [21] Video synopsis generation Activity, collision, and temporal consistency cost ✗

Authors	Motive	Optimized parameters	Classification
Acha et al.	Low level Video Synopsis Generation	Activity, occlusion, and length cost	✗
Pritch et al. [13]	Video Synopsis	Activity, collision, and temporal consistency cost	✗
Pritch et al. [14]	Video Synopsis and Video Indexing	Activity and discontinuity cost	✗
Pritch et al. [15]	Non-chrono-logical Video Synopsis	Motion and collision cost	✓
Tian et al. [16]	Video Synopsis	Activity, the cost-related positive sequence between objects, false collision and real collision cost	✓
Yi et al. [17]	Video Synopsis	Temporal model, density model and collision model cost	✗
Ruan et al. [18]	Online Video Synopsis	Tube collision, activity completeness and interaction preservation cost	✗
Ahmed et al. [19]	Intelligent Traffic Monitoring	Activity, collision and temporal consistency cost	✓
Ghatak et al. [20]	Video Synopsis	Activity, collision, and temporal consistency cost	✗
Ghatak et al. [21]	Video synopsis generation	Activity, collision, and temporal consistency cost	✗

2.2 360° video summarization

Su et al. proposed a novel Pano2Vid framework to generate a natural-looking NFoV for panoramic 360° videos [5]. For the given input 360° video, Pano2Vid generates glimpses like traditional viewing with a horizontal spanning angle of 65.5°. Pano2Vid uses a data-driven approach, namely, AutoCam. AutoCam uses dynamic programming for selecting optimal camera trajectories to generate glimpses. Pano2Vid compiles a new dataset for automatic cinematography in 360° videos. It uses 18 azimuthal and 11 polar angles. Su et al. adds additional capability to the Pano2Vid task by controlling its FoV dynamically using a coarse-to-fine optimization technique [6]. In this work, Pano2Vid was extended to Zoom Lens Pano2Vid with three horizontal spanning angles as {104.3°, 65.5°, 46.4°}. It uses 18 azimuthal and 11 polar angles.∥Hu et al. [7] proposed a solution for automatic 360° video piloting by using an online agent. With the help of the previously selected viewing angle and the key object, the future viewing angle will be predicted using a Recurrent Neural Network(RNN). For evaluation, a 360° Sports video dataset comprising five sports areas was introduced. 360° video spatial and temporal summarization was introduced by Yu et al. [8]. Composition View Score (CVS), a deep learning-based ranking approach, was proposed to identify which view is appropriate for 360° video highlight. CVS generates a score map of composition per video segment in spherical form. The CVS model outperforms the Pano2Vid dataset. It uses 4 azimuthal and 3 polar angles.∥Lee et al. presented a story based temporal 360° video summarization [9]. It uses a novel approach to the memory network model, Past Future Memory Network (PFMN), where the past memory stores selected sub-shots and the future memory consists of possible future sub-shots. It uses 9 azimuthal and 9 polar angles.∥Xu et al. suggested an approach to apply deep reinforcement learning-based approach to predict the viewports [25]. It extracts the spatio-temporal features of attention-related informations. It provides a database collecting head movements in panoramic videos. The proposed method was implemented in both online and offline versions. In the online approach, the future head movement is estimated based on the current head position, whereas the offline approach determines the head movement at each spherical frame. Then, a heat map was generated to predict the final head movement.∥Ban et al. performs prediction of future viewports through the viewer’s personalized and cross viewers behavior information [26]. A Quality of Experience (QoE) based framework is also introduced to optimize available video streaming methodologies and an algorithm to solve the Nondeterministic Polynomial (NP) problem at a low cost. The saliency computed at different scales is different; hence, Xu et al. solved the problem of tracking large-scale dynamic spherical videos [17]. Saliency maps are computed at different spatial scales: sub-image patch centered at the current viewpoint, sub-image corresponding to FoV, and spherical image. The saliency maps and the corresponding images are now given into the Convolutional Neural Network (CNN) for feature extraction. Long Short-Term Memory (LSTM) is used to encode the historical viewing path. Then the features of CNN and LSTM are combined to predict the gazes points. It uses 208 spherical videos with at least 31 subjects.∥Pietrangelo et al. introduces an algorithm for the long-term viewport prediction in the spherical videos [28]. The trajectories with similar viewing experiences are grouped, and a different function is used for each group. At each run-time, the pre-computed functions are used. In the field of viewport prediction, Wu et al. provided a solution to various challenges by using a preference-based viewport prediction network [29]. The 360 feature extraction is improved by using spherical CNN (S2CNN). A RNN is used to extract content information to utilize the spatial attention of the interested regions to achieve multimodal data learning. Tang et al. jointly exploited the viewport trajectory with multi-object tracking and the historical viewport trajectory for predicting the future viewport [30]. A multi-object selection procedure was introduced due to the multiple objects in the panoramic video. The proposed methodology supports multiple viewport prediction, whereas the existing works predicted only the single viewport.∥Chen et al. suggested a neural network architecture for predicting the future head movement [31]. The video features are extracted using CNN, and the future head movement is predicted using LSTM. It provides better prediction accuracy considering only the video’s position data. Chao et al. proposed a transformer-based viewport prediction in the spherical videos [32]. The suggested approach was implemented on three widely-used datasets and was observed to outperform the state-of-the-art methods in terms of computation complexity. It also provides high accuracy for long and short-term viewport prediction.∥Li et al. used S2CNN instead of a traditional two-dimensional CNN to mitigate the weight-sharing failure caused due to the video projection distortion [33]. Salient spatio-temporal features are extracted using a spherical convolution-based saliency detection model. RNN is used to predict the video’s time series of FoV information. Then the salient features and the time series information are combined to predict the future viewer’s head movement. Table 2 illustrates the existing works on the summarization of 360° videos.

Table 2
Survey on 360° Video Summarization

Auth-ors Model Technique Algorithm FOMO

Su et al. AutoCam Spatial Logistic Regression ✓

Su et al. [6] AutoCam with zooming Spatial Logistic Regression ✓

Hu et al. [7] Deep 360pilot Spatial RNN ✓

Yu et al. [8] CVS Spatial + Temporal Deep CNN ✓

Lee et al. [9] PFMN Spatial + Temporal Kernel Temporal Segmentation ✓

Xu et al. [25] Both offline and online prediction technique Spatial Deep reinforcement learning ✓

Ban et al. [26] Cross-users behaviors based viewport prediction Spatial K-Nearest-Neighbors ✓

Xu et al. [27] Gaze prediction Spatial CNN and LSTM ✓

Petrangeli et al. [28] Long-term prediction algorithm Spatial Trajectory clustering ✓

Wu et al. [29] Viewport prediction based on attention Spatial RNN ✓

Tang et al. [30] Long term viewport prediction Spatial LSTM ✓

Chen et al. [31] Content aware based viewport prediction Spatial CNN and LSTM ✓

Chao et al. [32] Long term viewport prediction method Spatial 360° viewport prediction transformer ✓

Li et al. [33] Future FoV prediction Spatial Spherical convolution-empowered FoV prediction approach ✓

Auth-ors	Model	Technique	Algorithm	FOMO
Su et al.	AutoCam	Spatial	Logistic Regression	✓
Su et al. [6]	AutoCam with zooming	Spatial	Logistic Regression	✓
Hu et al. [7]	Deep 360pilot	Spatial	RNN	✓
Yu et al. [8]	CVS	Spatial + Temporal	Deep CNN	✓
Lee et al. [9]	PFMN	Spatial + Temporal	Kernel Temporal Segmentation	✓
Xu et al. [25]	Both offline and online prediction technique	Spatial	Deep reinforcement learning	✓
Ban et al. [26]	Cross-users behaviors based viewport prediction	Spatial	K-Nearest-Neighbors	✓
Xu et al. [27]	Gaze prediction	Spatial	CNN and LSTM	✓
Petrangeli et al. [28]	Long-term prediction algorithm	Spatial	Trajectory clustering	✓
Wu et al. [29]	Viewport prediction based on attention	Spatial	RNN	✓
Tang et al. [30]	Long term viewport prediction	Spatial	LSTM	✓
Chen et al. [31]	Content aware based viewport prediction	Spatial	CNN and LSTM	✓
Chao et al. [32]	Long term viewport prediction method	Spatial	360° viewport prediction transformer	✓
Li et al. [33]	Future FoV prediction	Spatial	Spherical convolution-empowered FoV prediction approach	✓

3 Proposed methods

The approach proposed in this work involves the following steps: First is the mathematical formulation of the energy minimization problem. The second is the generation and visualization of spherical video summaries.

3.1 Problem formulation

A spherical video synopsis (S) is obtained by shifting the objects in the recorded spherical surveillance video (r). This work concentrates on the temporal shifting of the objects, keeping the spatial domain unchanged. Based on the literature related to traditional video synopsis, the Equation 1 was formulated. Here, a new constraint namely display cost was integrated. The temporal shift (M) is achieved by minimizing the subsequent energy cost function: $\begin{matrix} E (M) = β_{1} E_{A} (M) + β_{2} E_{c} (M) + β_{3} E_{T} (M) \\ + β_{4} E_{L} (M) + β_{5} E_{D} (M) \end{matrix}$ (1) where E_A (M) denotes the activity cost, E_C (M) denotes the collision cost, E_T (M) denotes the temporal consistency cost, E_L (M) denotes the length cost and E_D (M) denotes the display cost while β₁,β₂,β₃,β₄,β₅ are the weight that indicates the importance of the corresponding cost.

Activity cost

It penalizes for the activity that is missed out while mapping, r to S. It is zero if all the objects (O) in r are mapped to S [12]. $\begin{matrix} E_{A} (M) = \sum E_{A} (O) = \sum_{i = 1}^{n} o_{i} \end{matrix}$ (2) where o_i is the object track that are missed out and n is the total number of tracks.

Collision cost

It adds a penalty for the collision of two temporally shifted objects in S [12]. $\begin{matrix} E_{C} (M) = \sum E_{C} (O_{x}, O_{y}) \\ = \sum Area (bbox (O_{x}) \cap bbox (O_{y})) \end{matrix}$ (3) where O_x,O_y denotes the two temporally shifted objects x and y in S. Area function computes the area of intersection between two shifted objects O_x,O_y. And, bbox denotes the bounding box of the shifted objects O_x,O_y. The collision cost will be zero if there is no collision between the two shifted objects.

Temporal consistency cost

It adds a penalty for violating the chronological order of events that are shifted in S. If there is no violation, then it will be zero [13]. $\begin{matrix} E_{T} (M) = \sum_{i = 1}^{n} | order (r_{O_{i}} - S_{O_{i}}) | \end{matrix}$ (4) where, r_{O
_i} and S_{O
_i} denotes the object tubes O_i of r and S respectively.

Length cost

It penalizes for a lengthy synopsis in comparison with r [3]. $\begin{matrix} E_{L} (M) = length (S) \end{matrix}$ (5)

Display cost

The human visual system is restricted to handle a fixed number of objects less than eight at the same time [10]. The existing traditional video synopsis frameworks have ignored this concept. It penalizes for showing more than the viewer’s limited number of objects per frame in S. $\begin{matrix} E_{D} (M) = \sum_{i = 1}^{m} O_{i} (t) \end{matrix}$ (6) where m denotes the total number of objects per frame t in S. In this work, the number of objects shown utmost per frame is seven.

3.2 Generation and visualization of spherical video synopsis

The pipeline for generating and visualizing spherical video synopsis consists of four phases: recording, processing, stitching, and visualization. This is shown in Fig. 2.

Fig. 2

Phases involved in the generation and visualization of spherical synopsis video

3.2.1 Recording phase

The 360° surveillance video is recorded using a spherical camera. Here, the scene is recorded using multiple cameras which are independent of each other and are stitched together to create a spherical view. The frame of the recorded 360° surveillance video in equirectangular projection is given in Fig. 3.

Fig. 3

Frame No: 4574 extracted from the recorded spherical surveillance video in equirectangular projection

3.2.2 Processing phase

In this phase, spherical background extraction, object detection, and track extraction are performed. The background of r with no moving objects is extracted by using the time-lapse background video generation [14]. It uses a temporal histogram. Figure 4 illustrates a time-lapse background extracted from the recorded spherical surveillance video.

Fig. 4

Illustration of a time-lapse background extracted from a recorded spherical surveillance video

Object detection and track extraction are the key tasks in generating an object-based video synopsis for spherical surveillance video. Faster R-CNN [35] is used to detect moving objects in r. Figure 5 illustrates object detection performed on recorded spherical video. Once the objects are detected, the track extraction is done by using Deep SORT [36]. A visualization of the 12 object track paths denoting the starting and end point in the input video is illustrated in Fig. 6. The object tube consists of a sequence of activities performed by the respective objects throughout the recorded surveillance video. Finally, extracted tracks and the objects’ respective timestamps are placed individually in an object store.

Fig. 5

Object detection performed on recorded spherical video

Fig. 6

A visualization of the 12 object track paths extracted from the recorded spherical video

The interactions between two moving objects namely, O_A and O_B in r are combined into a single tube and preserved in S. Examples of interactions include two people talking for a few minutes, a handshake, and an exchange of items. The recursive tube-grouping algorithm [37] is used to group such interactive objects.

3.2.3 Stitching phase

The objects in the object store undergo the energy minimization process to minimize Equation (1). According to the best temporal positions obtained, the objects and their respective timestamps are stitched to the extracted spherical background image using Poisson image editing [38] in the stitching phase. Figure 7 represents the stitched objects with timestamps to the spherical background.

Fig. 7

Representation of the stitched objects with timestamps to the extracted background

3.2.4 Visualization phase

360° video offers an immersive experience to the viewers. However, if the viewer is looking at one part, the viewer is more likely to miss an event occurring in the other part of the sphere. It can be commonly termed as FOMO [39] as it contains multidirectional information about the scene. Therefore, it is important to view all the scenes ignoring irrelevant content (i.e., giving less viewing importance to insignificant content). As a result, the audience can see and interpret all the key parts of the content that needs to be viewed without FOMO. Visual saliency prediction allows the viewer to focus on certain locations or parts of the visual scene while ignoring other information based on its importance. Graph-Based Visual Saliency 360 (GBVS360) [34] is used to generate a spherical saliency map. Figure 8 illustrates the saliency map of the recorded spherical surveillance video. It provides a probability distribution of human visual attention within a spherical frame.

Fig. 8

Illustration of recorded 360° surveillance video saliency map

In this work, it is observed that the top and bottom scenes cover less salient information, hence these two regions are given less importance. Now, the viewport is generated for other regions using rectilinear projection. It maps the spherical portion to a flat space [40]. The viewport generation is primarily dependent on the pitch, yaw, and roll angles. The pitch and yaw angle (θ, φ) are plotted to the row and column of Cartesian coordinate (x, y) respectively. As per the literature, the roll angle is set at 0° [41]. The existing works on 360° video summarization uses at least 12 NFoV glimpses. Therefore, in this work, the condensed spherical video synopsis is visualized in multiple directions as per the traditional videography having limited FoV of less than 12 NFoV glimpses i.e., θ ={ 0° } and φ ={ 0°, ± 90°, 180° }. This allows all salient areas to be displayed in the video synopsis while ignoring less important areas in multiple directions at the same time (i.e., direction-based visualization). Visualizing the scene from multiple directions simultaneously eliminates FOMO from the generated video synopsis.

4 Experimental results and performance analysis

Due to the unavailability of real-time spherical surveillance video, the user-generated video was recorded from the National Institute of Technology Puducherry at 24 fps with Insta360 ONE X. The video duration (Z) is 01:03:11 (HH:MM:SS), and it contains 110 360° objects with 20 interacting objects. The objects include pedestrians, two-wheelers, and four-wheelers. The resolution of the recorded spherical frame and FoV of generated synopsis video is medmuskip = 0mu5760 × 2880 and medmuskip = 0mu1200 × 1200 respectively. The number of frames of the recorded 360° surveillance video is 91,000. Exhaustive experiments are carried out on the recorded spherical surveillance videos for the performance analysis of the proposed framework. During the development and testing phase of the proposed approach, MATLAB (version 2019a) with an Intel Core i7 2.60 GHz CPU with 16GB RAM is employed. Figure 9 presents the viewport generation performed by the rectilinear projection of a 360° surveillance synopsis video. The preservation of interacting objects in the synopsis video is shown in Fig. 10. To track an object from one area to another, a direction-based video synopsis of each area must be played back simultaneously. Figure 11 gives the multiple direction-based video synopsis of 35^th frame, where each area is played synchronously.

The proposed work on the generation of spherical surveillance synopsis video experiments on two cases:

Varying synopsis length (V_L)

In this case, the length of the spherical synopsis video will vary for each optimization algorithm based on its performance.

Fixed synopsis length (F_L)

In this case, the length of the spherical synopsis video is the longest object tube length for all optimization algorithms.

Fig. 9

Viewport generation performed using the rectilinear projection of a 360° surveillance synopsis video

Fig. 10

Preservation of interacting objects in the synopsis video

Fig. 11

Multiple direction-based synopsis of Frame No: 35, where each area is played synchronously with only 4 NFoV glimpses

4.1 Performance analysis

The proposed spherical synopsis framework has experimented with recent metaheuristics optimization algorithms such as follows, Ant Lion Optimization (ALO) [44], Aquila Optimization (AO) [45], Giza Pyramids Construction (GPC) [46], Grey Wolf Optimization (GWO) [47], Heap-Based Optimization (HBO) [48], Hybrid of Particle Swarm and Grey Wolf Optimization (HPSOGWO) [49] for performance analysis. It is also compared with optimization algorithms used in existing traditional video synopsis works such as Hybrid of Grey Wolf Optimization and Simulated Annealing (HGWOSA) [21], HSAJAYA [50], Hybrid of Simulated Annealing and Teacher Learning based Optimization (HSATLBO) [20], Particle Swarm Optimization (PSO) [42], and Simulated Annealing (SA) [12].

Compared to other costs, collision costs have a higher weight to effectively reduce the collision of 360° objects. The activity, collision, temporal consistency, length, and display cost weights are 0.1, 0.4, 0.2, 0.1, and 0.2, respectively. 100 iterations are used for the optimization algorithms, and 10 populations are used. The assignment of other specific parameter values for tuning is the same as that described in the relevant literature.

Here, the activity cost is zero in both cases, which indicates that the object’s loss is zero. Figure 12 illustrates the convergence characteristics of the used optimization algorithms. It is observed that compared with other optimization algorithms, ALO [44] and PSO [42] for V_L and ALO [44] for F_L converges in less time compared with other optimization algorithms.

Fig. 12

Analysis of convergence characteristics for both V_L and F_L

Table 3 presents the experimental analysis of the optimization algorithms used in the proposed framework for V_L. According to Table 3, we can reasonably infer as follows,

Video synopsis generated by optimization algorithms other than ALO [44] and HBO [48] shifts the objects with high collision.

Video synopsis generated by optimization algorithms other than GPC [46] shifts the object with a high temporal disorder.

Video synopsis generated using PSO [42] has minimum synopsis length.

Video synopsis generated using HGWOSA [21] packs the desired number of objects per frame effectively.

Table 3

Experimental analysis of the proposed framework for V_L

	Algorithm	E_C	E_T	E_L	E_F
Proposed	ALO [44]	0.0090	0.0200	0.0122	0.2500
	AO [45]	0.0098	0.0400	0.0126	0.2420
	GPC [46]	0.0097	0.0100	0.0408	0.2460
	GWO [47]	0.0102	0.0700	0.0522	0.2630
	HBO [48]	0.0090	0.0200	0.0244	0.0980
	HPSOGWO [49]	0.0100	0.0800	0.0267	0.2730
Existing	HGWOSA [21]	0.0222	0.0211	0.0372	0.0104
	HSAJaya [50]	0.0096	0.0600	0.0244	0.2760
	HSATLBO [20]	0.0284	0.0800	0.0133	0.0400
	PSO [42]	0.0244	0.1000	0.0036	0.0421
	SA [12]	0.0323	0.0800	0.0122	0.0187

Table 4 presents the experimental analysis of the optimization algorithms used in the proposed framework for P_L. According to Table 4, we can reasonably infer as follows,

Video synopsis generated by optimization algorithms other than SA [12] shifts the objects with high collision.

Video synopsis generated by optimization algorithms other than GPC [46] and HBO [48] shifts the object with a high temporal disorder.

Video synopsis generated using HSATLBO [20] packs the desired number of objects per frame effectively.

Table 4

Experimental analysis of the proposed framework for P_L

	Algorithm	E_C	E_T	E_F
Proposed	ALO [44]	0.0290	0.0600	0.0119
	AO [45]	0.0278	0.0300	0.0230
	GPC [46]	0.0283	0.0100	0.0170
	GWO [47]	0.0294	0.0800	0.0590
	HBO [48]	0.0278	0.0100	0.0550
	HPSOGWO [49]	0.0394	0.1800	0.1000
Existing	HGWOSA [21]	0.0274	0.0245	0.0512
	HSAJaya [50]	0.0278	0.0300	0.0750
	HSATLBO [20]	0.0264	0.0400	0.0081
	PSO [42]	0.0315	0.0400	0.0106
	SA [12]	0.0242	0.0900	0.0221

The evaluation metrics used are object loss, activity preservation rate, space efficiency, chronological disorder ratio, and overall virtual collision rate.

Object loss

It is the ratio of the number of objects in the synopsis video (S_O) to the recorded 360° surveillance video r_O [16]. It is denoted as σ. $\begin{matrix} σ = 1 - \frac{S_{O}}{r_{O}} \end{matrix}$ (7)

Activity preservation rate

It is the ratio of the number of activities in the synopsis video (A_S) to the recorded video (A_r) [51]. It is referred to as η. $\begin{matrix} η = \frac{A_{S}}{A_{r}} \times 100 \end{matrix}$ (8)

Space efficiency

Space efficiency (S_E) is defined as the ratio of the number of frames discarded by the synopsis (f_d) to the number of frames (N) in the recorded video [43]. $\begin{matrix} S_{E} = \frac{f_{d}}{N} \times 100 \end{matrix}$ (9)

Chronological disorder ratio

It is defined as the ratio between the sum of all disordered objects (O_d) to the total number of moving objects (T_Mo) [37]. Disordered objects mean objects that are not following the order of appearance as that of the recorded 360° surveillance video. It is denoted as δ. $\begin{matrix} δ = \frac{O_{d}}{T_{Mo}} \end{matrix}$ (10)

Overall virtual collision

It is denoted as χ and given as follows [37], $\begin{matrix} χ (a, b) = {\begin{matrix} V & if k_{a}^{S} - k_{b}^{S} = k_{\hat{a}}^{S} - k_{\hat{b}}^{S} \\ {AI}_{S} (a, b) & otherwise \end{matrix}} \end{matrix}$ (11) where, V is AI_S (a, b) - AI_r (a, b), AI_S (a, b) and AI_r (a, b) are the area of intersection between the two tubes a and b in S and r respectively. $\begin{matrix} AI (a, b) \\ = \sum_{F ɛ F_{a} \cap F_{b}} Area (bbox (a^{F}) \cap bbox (b^{F})) \end{matrix}$ (12) where Area (bbox (a^F) ∩ bbox (b^F)) determines the area in the 360° frame F that is common to both tube a and b. $k_{a}^{S}$ and $k_{b}^{S}$ denotes the starting time of the tubes a and b in r while $k_{\hat{a}}^{S}$ and $k_{\hat{b}}^{S}$ denotes the starting time of the tubes a and b in S.

The performance metrics of the spherical synopsis video with V_L are given in Table 5. PSO [42] generates a spherical synopsis with minimum length and maximum space efficiency. Better chronological disorder ratio and overall virtual collision are provided by GPC [46] and HBO [48] respectively.

Table 5

Performance metrics of the spherical synopsis with V_L

	Algorithm	N	Z	S_E(%)	δ	χ
Proposed	ALO [44]	30,096	20:90	66.93	0.0386	0.0112
	AO [45]	30,097	20:90	66.93	0.0407	0.0120
	GPC [46]	36,121	25:08	60.31	0.0355	0.0117
	GWO [47]	37,814	26:26	58.45	0.0436	0.0143
	HBO [48]	33,669	23:38	63.00	0.0386	0.0011
	HPSOGWO [49]	33,941	23:57	62.70	0.1279	0.0137
Existing	HGWOSA [21]	35,028	24:33	61.51	0.0391	0.0158
	HSAJAYA [50]	33,669	23:38	63.00	0.0418	0.0114
	HSATLBO [20]	30,099	20:90	66.92	0.0455	0.0193
	PSO [42]	27,720	19:25	69.54	0.1727	0.0169
	SA [12]	30,096	20:90	66.93	0.1273	0.0207

Table 6 gives the performance metrics used in the proposed work for generating a fixed-length synopsis. For F_L, the synopsis length is 17,345, equal to the longest object tube length in the recorded surveillance video, and the synopsis duration Z is 00:12:02. The space efficiency is 80.94%. The synopsis generated using HBO [48] and SA [12] provide better chronological disorder ratio and overall virtual collision results, respectively. For both cases V_L and F_L, all the objects in the recorded video are included in the synopsis video, thus making the object loss and activity preservation rate 0 and 100% respectively.

Table 6

Performance metrics of the spherical synopsis with F_L

	Algorithm	δ	χ
Proposed	ALO [44]	0.2745	0.3181
	AO [45]	0.2231	0.3047
	GPC [46]	0.2141	0.3094
	GWO [47]	0.2876	0.3258
	HBO [48]	0.2012	0.2999
	HPSOGWO [49]	0.2942	0.3478
Existing	HGWOSA [21]	0.2162	0.2957
	HSAJAYA [50]	0.2189	0.2982
	HSATLBO [20]	0.2273	0.2931
	PSO [42]	0.2273	0.3457
	SA [12]	0.2909	0.2914

Table 7 presents the analysis of the activity preserved for the viewer-specified video duration for V_L and F_L for 4 minutes. Here, ALO [44] provides better results for both object loss and activity preservation rate for both V_L and F_L. Table 8 presents the analysis of the activity preserved for the viewer-specified video duration for V_L and F_L for 6 minutes. For object loss in V_L and F_L, ALO [44] and AO [45] performs better, respectively. For activity preservation rate in V_L and F_L, ALO [44] and AO [45] performs better, respectively.

Table 7

Analysis on the activity preserved for the viewer specified video duration for V_L and F_L for 4 minutes

	Algorithm	σ		η (%)
		V_L	F_L	V_L	F_L
Proposed	ALO [44]	0.3636	0.5000	63.64	50.00
	AO [45]	0.7909	0.5182	20.91	48.18
	GPC [46]	0.9273	0.6364	07.27	36.36
	GWO [47]	0.9091	0.8545	09.09	14.55
	HBO [48]	0.8091	0.6273	19.09	37.27
	HPSOGWO [49]	0.8273	0.7364	17.27	26.36
Existing	HGWOSA [21]	0.6364	0.6091	36.36	39.09
	HSAJAYA [50]	0.6818	0.7636	31.82	23.64
	HSATLBO [20]	0.6182	0.5273	38.18	47.27
	PSO [42]	0.6636	0.6636	33.64	33.64
	SA [12]	0.5364	0.6364	46.36	36.36

Table 8

Analysis of the activity preserved for the viewer specified video duration for V_L and F_L for 6 minutes

	Algorithm	σ		η(%)
		V_L	F_L	V_L	F_L
Proposed	ALO [44]	0.3273	0.3909	67.27	60.91
	AO [45]	0.5364	0.3636	46.36	63.64
	GPC [46]	0.9000	0.5545	10.00	44.55
	GWO [47]	0.8545	0.7455	14.55	25.45
	HBO [48]	0.7000	0.5182	30.00	48.18
	HPSOGWO [49]	0.7545	0.6545	24.55	34.55
Existing	HGWOSA [21]	0.4091	0.5455	59.09	45.45
	HSAJAYA [50]	0.5545	0.6091	44.55	39.09
	HSATLBO [20]	0.6000	0.4455	40.00	55.45
	PSO [42]	0.5364	0.6000	46.36	40.00
	SA [12]	0.4273	0.5091	57.27	49.09

Analysis of the activity preserved for the viewer-specified video duration for V_L and F_L for 8 minutes is given in Table 9. Here, for object loss in V_L and F_L, ALO [44] and AO [45] performs better, respectively. For activity preservation rate in V_L and F_L, ALO [44] and AO [45] performs better, respectively.

Table 9

Analysis on the activity preserved for the viewer specified video duration for V_L and F_L for 8 minutes

	Algorithm	σ		η (%)
		V_L	F_L	V_L	F_L
Proposed	ALO [44]	0.2182	0.3364	78.18	66.36
	AO [45]	0.4091	0.3091	59.09	69.09
	GPC [46]	0.8545	0.4455	14.55	55.45
	GWO [47]	0.8273	0.6455	17.27	35.45
	HBO [48]	0.6091	0.4636	39.09	53.64
	HPSOGWO [49]	0.6727	0.5364	32.73	46.36
Existing	HGWOSA [21]	0.3727	0.3818	62.73	61.82
	HSAJAYA [50]	0.5000	0.4636	50.00	53.64
	HSATLBO [20]	0.5455	0.4182	45.45	58.18
	PSO [42]	0.5091	0.5636	49.09	43.64
	SA [12]	0.3182	0.4364	68.18	56.36

Table 10 presents the analysis of the activity preserved for the viewer-specified video duration for V_L and F_L for 10 minutes. Here, ALO [44] provides better results for object loss and activity preservation rate.

Table 10

Analysis of the activity preserved for the viewer specified video duration for V_L and F_L for 10 minutes

	Algorithm	σ		η (%)
		V_L	F_L	V_L	F_L
Proposed	ALO [44]	0.1636	0.1100	83.64	80.91
	AO [45]	0.3364	0.1700	66.36	75.45
	GPC [46]	0.7455	0.3300	25.45	60.91
	GWO [47]	0.7818	0.4800	21.82	47.27
	HBO [48]	0.4909	0.3400	50.91	60.00
	HPSOGWO [49]	0.5545	0.3200	44.55	61.82
Existing	HGWOSA [21]	0.3182	0.2800	68.18	65.45
	HSAJAYA [50]	0.3727	0.3700	62.73	57.27
	HSATLBO [20]	0.5273	0.3000	47.27	63.64
	PSO [42]	0.4545	0.4800	54.55	47.27
	SA [12]	0.2000	0.2900	80.00	64.55

Table 11 presents the analysis of the activity preserved for the viewer-specified video duration for V_L and F_L for 12 minutes. Here, for object loss in V_L, ALO [44] and SA [12] while, in F_L, ALO [44] performs better. For activity preservation rate in V_L, ALO [44] and SA [12] while, in F_L, ALO [44] performs better.

Table 11

Analysis of the activity preserved for the viewer specified video duration for V_L and F_L for 12 minutes

	Algorithm	σ		η (%)
		V_L	F_L	V_L	F_L
Proposed	ALO [44]	0.1364	0.0900	86.36	90.91
	AO [45]	0.2000	0.1091	80.00	89.09
	GPC [46]	0.7091	0.2909	29.09	70.91
	GWO [47]	0.7182	0.3727	28.18	62.73
	HBO [48]	0.3636	0.3182	63.64	68.18
	HPSOGWO [49]	0.4909	0.3636	50.91	63.64
Existing	HGWOSA [21]	0.3000	0.2727	70.00	72.73
	HSAJAYA [50]	0.2727	0.1818	72.73	81.82
	HSATLBO [20]	0.4545	0.3364	54.55	66.36
	PSO [42]	0.4273	0.4364	57.27	56.36
	SA [12]	0.1364	0.2273	86.36	77.27

An analysis of the amount of activity held within the viewer-specified synopsis duration is performed to see how the optimization algorithm works regarding activity preservation rate. Figure 13 illustrates the analysis of the activity preserved for the viewer-specified synopsis video duration for V_L and F_L using recent metaheuristics optimization algorithms like ALO [44], AO [45], GPC [46], GWO [47], HBO [48], HPSOGWO [49] and compared with the existing traditional video synopsis works such as HGWOSA [21], HSAJAYA [50], HSATLBO [20], PSO [42], and SA [12].

Fig. 13

Analysis of the activity preserved for the viewer-specified video duration (i.e., 4 minutes, 6 minutes, 8 minutes, 10 minutes, and 12 minutes) for V_L and F_L

To understand the effectiveness of the new display constraint in the proposed framework a user rating is performed to assess the human visual system experience of the user for the generated video synopsis. Here, 10 users are selected for rating purposes. Each user is requested to rate the individual algorithm out of 10 for three cases such as 6, 8, and 10 objects per frame. Table 12 presents the average user rating for the mentioned cases. It is observed that 8 objects per frame offer the human visual system a better opportunity to understand the scene perfectly.

Table 12

User rating based on the number of objects displayed per frame

Number of objects displayed	6	8	10	12
Average user rating (%)	66.36	83.63	23.63	8.18

A comparative statistical analysis was performed to validate the effectiveness of the proposed framework over the other considered meta-heuristic methods presented in Table 13 in terms of best, mean, worst, standard deviation (S.D), and execution time in seconds. It is observed that recent metaheuristics optimization algorithm yields better results compared to the existing traditional video synopsis work.

Table 13

Statistical comparison to validate the effectiveness of the proposed framework

	Algorithm	Best	Mean	Worst	S.D	Time
Proposed	ALO [44]	70.10	50.42	88.04	27.92	120.45
	AO [45]	326.40	486.27	994.01	297.43	74.12
	GPC [46]	70.17	429.94	875.21	272.14	50.17
	GWO [47]	20.25	44.31	98.14	29.61	90.12
	HBO [48]	16.57	518.55	890.10	295.57	40.18
	HPSOGWO [49]	02.49	47.10	105.32	27.98	60.91
Existing	HGWOSA [21]	33.64	50.44	124.01	30.16	80.17
	HSAJAYA [50]	591.15	547.53	875.04	282.13	112.41
	HSATLBO [20]	658.21	514.11	856.78	286.66	89.01
	PSO [42]	11.63	52.19	79.34	30.11	94.15
	SA [12]	40.21	49.04	101.12	29.15	89.23

A corner case analysis is performed to ensure the scalability of the proposed framework. This is shown in Table 14. Two test cases were used and it was observed that both test cases passed the proposed framework. This shows that the proposed framework is effectively scalable. And the proposed framework has been tested on different types of spherical video. It is found that the proposed approach serves as a generalized framework for creating spherical video synopsis.

Table 14

Corner case analysis for the proposed framework

Test Case ID	Description	Expected Result	Actual Result	Status
1.	To check when there are no objects.	No generation of video synopsis.	Optimization algorithms are not initiated to generate video synopsis and hence no video synopsis is generated.	Pass
2.	To check when there occurs many objects.	Generation of video synopsis with minimum collision and minimum temporal consistency.	Optimization algorithms generates video synopsis by minimizing collision and minimizing temporal consistency.	Pass

5 Conclusion

This paper proposes a novel spherical surveillance video synopsis generation and visualization framework with interaction preservation. The main contribution of this work is the elimination of FOMO through direction-based visualization by using only four NFoV glimpses. To better understand the scene, the framework allows the flexibility to restrict the objects the viewer sees in each frame. A comprehensive comparative analysis of recent metaheuristic optimization algorithms and existing video synopsis work is performed for varying and fixed synopsis lengths. The analysis is also done based on the viewer’s input constraints on the length of the synopsis video. Various performance metrics are used to find the efficacy of the proposed framework. Experimental analysis suggests that the proposed framework provides better results in solving the multiple constraints through the utilization of recent metaheuristic optimization algorithms compared with the state-of-art methods in the domain of video synopsis. The limitation of this work is that it forces the viewer to visualize from multiple directions. This is a difficult task as the viewer has to track single object in the video synopsis in multiple directions simultaneously.

References

Priyadharshini

and Mahapatra

Ansuman

, Panosyn: immersive video synopsis for spherical surveillance video, Sadhana 47(3) (2022),1–20.

Drisya

S.S.

Mahapatra

Ansuman

and Priyadharshini

, 360-degreeimage classification and viewport prediction using deep neural networks. In Advances in Distributed Computing and Machine Learning, pp. 483–492. Springer, 2022.

Mahapatra

Ansuman

, Sa

K P

, Majhi

Banshidhar

and Padhy

Sudarshan

, MVS: A multi-view video synopsis framework, Signal Processing: Image Communication. 42 (2016), 31–44.

Priyadharshini

and Mahapatra

Ansuman

, 360°user-generated videos: Current research and future trends, High Performance Vision Intelligence: Recent Advances 913 (2020), 117.

Y.C.

, Jayaraman

and Grauman

, Pano2vid: Automatic cinematography for watching 360° videos, Asian Conferenceon Computer Vision, Springer, pp. 154–171, 2016.

Y.C.

and Grauman

, Making 360 video watchable in 2d: Learning videography for click free viewing, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 1368–1376, 2017.

H.N.

, Lin

Y.C.

, Liu

M.Y.

, Cheng

H.T.

, Chang

Y.J.

and Sun

, Deep360 pilot: Learning a deep agent for piloting through 360 sportsvideos, 2017 IEEE Conference on Computer Vision and PatternRecognition (CVPR), IEEE, pp. 1396–1405, 2017.

, Lee

, Na

, Kang

and Kim

, A deep ranking model forspatio-temporal highlight detection from a 360° video, Proceedings of the AAAI Conference on Artificial Intelligence 32 (2018).

Lee

, Sung

, Yu

and Kim

, A memory network approach forstory-based temporal summarization of 360 videos, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), 1410–1419.

10.

Alvarez

G.A.

and Franconeri

S.L.

, How many objects can you track?: Evidence for a resource-limited attentive tracking mechanism, Journal of Vision 7(13) (2007),14–14.

11.

Priyadharshini

and Mahapatra

Ansuman

, A personalized video synopsis framework for spherical surveillance video, Computer Systems Science and Engineering.

12.

Acha

A.R.

, Pritch

and Peleg

, Making a long video short: Dynamic video synopsis. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 1, pp. 435–441. IEEE, 2006.

13.

Pritch

, Acha

A.R.

, Gutman

and Peleg

, Webcam synopsis:Peeking around the world, In 2007 IEEE 11th InternationalConference on Computer Vision. pp. 1–8. IEEE, 2007.

14.

Pritch

, Acha

A.R.

and Peleg

, Nonchronological video synopsisand indexing, IEEE Transactions on Pattern Analysis and Machine Intelligence 30(11) (2008), 1971–1984.

15.

Pritch

, Ratovitch

, Hendel

and Peleg

, Clustered synopsis of surveillance video, In 2009 Sixth IEEE international conference on advanced video and signal based surveillance. pp. 195–200. IEEE, 2009.

16.

Tian

, Zheng

, Chen

, Wang

and Lin

, Surveillance video synopsis generation method via keeping important relationship among objects, IET Computer Vision 10(8) (2016),868–872.

17.

, Jun

, Nong

, Zhiguo

and Changxin

, Chronological video synopsis via events rearrangement optimization, Chinese Journal of Electronics 27(2) (2018),399–404.

18.

Ruan

, Wei

, Li

and Zhao

, Rearranging online tubes for streaming video synopsis: A dynamic graph coloring approach, IEEE Transactions on Image Processing 28(8) (2019), 3873–3884.

19.

Ahmed

S.A.

, Dogra

D.P.

, Kar

, Patnaik

, Lee

S.C.

, Choi

, Nam

G.P.

and Kim

I.J.

, Query-based video synopsis for intelligent traffic monitoring applications, IEEE Transactions on Intelligent Transportation Systems, 21(8) (2019), 3457–3468.

20.

Ghatak

, Rup

, Majhi

and Swamy

M.N.

, An improved surveillance video synopsis framework: a hsatlbo optimization approach, Multimedia Tools and Applications 79(7) (2020), 4429–4461.

21.

Ghatak

, Rup

, Didwania

and Swamy

M.N.

, Gan based efficient foreground extraction and hgwosa based optimization for video synopsis generation, Digital Signal Processing 111(2021), 102988.

22.

Ghatak

and Rup

, Single camera surveillance video synopsis: areview and taxonomy, In 2019 International Conference on Information Technology (ICIT), pp. 483–488. IEEE, 2019.

23.

Baskurt

K.B.

and Samet

, Video synopsis: a survey, Computer Vision and Image Understanding. 181 (2019), 26–38.

24.

Mahapatra

and Sa

P.K.

, Video synopsis: A systematic review, High Performance Vision Intelligence (2020), 101–115.

25.

Mai

, Song

Yuhang

, Wang

Jianyi

, Qiao

Ming Lang

, Huo

Liangyu

and Wang

Zulin

, Predicting head movement in panoramic video: A deep reinforcement learning approach, IEEE Transactions on Pattern Analysis and Machine Intelligence 41(11) (2693), 2708.

26.

Ban

Yixuan

, Xie

Lan

, Xu

Zhimin

, Zhang

Xinggong

, Guo

Zongming

and Wang

Yue

, Cub360: Exploiting cross-users behaviors for viewport prediction in 360 video adaptive streaming, In 2018 IEEE International Conference on Multimedia and Expo (ICME). pages 1–6. IEEE, 2018.

27.

Yanyu

, Dong

Yanbing

, Wu

Junru

, Sun

Zhengzhong

, Shi

Zhiru

, Yu

Jingyi

and Gao

Shenghua

, Gaze prediction in dynamic 360 immersive videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), 5333–5342.

28.

Petrangeli

Stefano

, Simon

Gwendal

and Swaminathan

Viswanathan

, Trajectory-based viewport prediction for 360-degree virtual realityvideos. In 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR) IEEE (2018), 157–160.

29.

Chenglei

, Zhang

Ruixiao

, Wang

Zhi

and Sun

Lifeng

, A spherical convolution approach for learning long term viewport prediction in 360 immersive video. In Proceedings of the AAAI Conference on Artificial Intelligence 34 (2020), 14003–14040.

30.

Tang

Jinting

, Huo

Yongkai

, Yang

Shaoshi

and Jiang

Jianmin

, A viewport prediction framework for panoramic videos. In 2020 International Joint Conference on Neural Networks (IJCNN) (2020), IEEE 1–8.

31.

Chen

Xinwei

, Kasgari

Ali Taleb Zadeh

and Saad

Walid

, Deep learning for content-based personalized viewport prediction of 360-degree vr videos, IEEE Networking Letters 2(2) (2020),81–84.

32.

Chao

Fang-Yi

, Ozcinar

Cagri

and Smolic

Aljosa

, Transformer-based long-term viewport prediction in 360° video: Scan path isall you need. In MMSP (2021), 1–6.

33.

Jie

, Han

Ling

, Zhang

Chong

, Li

Qiyue

and Liu

Zhi

, Spherical convolution empowered viewport prediction in 360 video multicast with limited fov feedback, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) (2022), 1–16.

34.

Lebreton

and Raake

, GBVS360, BMS360, ProSal: Extending existing saliency prediction models from 2d to omnidirectional images, Signal Processing: Image Communication 69(2018), 69–78.

35.

Ren

, He

, Girshick

and Sun

, Faster r-cnn: Towards real-time object detection with region proposal networks, Advances in Neural Information Processing Systems. 28 (2015), 91–99.

36.

Wojke

, Bewley

and Paulus

, Simple online and realtime tracking with a deep association metric, 2017 IEEE international conference on image processing (ICIP), IEEE, pp. 3645–3649, 2017.

37.

Narayanan

, Preserving interactions among moving objects in surveillance video synopsis, Multimedia Tools and Applications 79(43) (2020),32331–32360.

38.

Pérez

, Gangnet

and Blake

, Poisson image editing, ACM SIGGRAPH 2003 Papers, pp. 313–318, 2003.

39.

Aitamurto

, Won

A.S.

, Sakshuwong

, Kim

, Sadeghi

, Stein

Royal

P.G.

and Kircos

C.L.

, From fomo to jomo: Examining the fearand joy of missing out and presence in a 360 video viewing experience. In Proceedings of the 2021 CHI Conference on HumanFactors in Computing Systems (2021), 1–14.

40.

, Alshima

and Boyce

, Jvet-e1003: Algorithm descriptions of projection format conversion and video quality metrics in 360lib, 07 2018.

41.

Bao

, Wu

, Zhang

, Ramli

A.A.

and Liu

, Shooting a moving target: Motion-prediction-based transmission for 360-degree videos, 2016 IEEE International Conference on Big Data (Big Data), IEEE, pp. 1161–1170, 2016.

42.

Moussa

M.M.

and Shoitan

, Object-based video synopsis approach using particle swarm optimization, Signal, Image and Video Processing 15(4) (2021), 761–768.

43.

Chou

C.L.

, Lin

C.H.

, Chiang

T.H.

, Chen

H.T.

and Lee

S.Y.

, Coherent event-based surveillance video synopsis using trajectory clustering. In 2015 IEEE international conference on multimedia & expoworkshops (ICMEW). IEEE (2015), 1–6.

44.

Ali

, Elazim

S.M. Abd.

and Abdelaziz

A.Y.

, Ant lion optimization algorithm for renewable distributed generations, Energy. 116 (2016), 445–458.

45.

Abualigah

, Yousri

, Elaziz

M. Abd.

, Ewees

A.A.

, Al.qaness

M.A.A.

and Gandomi

A.H.

, Aquila optimizer: A novel meta-heuristic optimization algorithm, Computers & Industrial Engineering 157 (2021), 107250.

46.

Harifi

, Mohammadzadeh

, Khalilian

and Ebrahimnejad

, Giza pyramids construction: an ancient-inspired metaheuristic algorithm for optimization, Evolutionary Intelligence (2020), 1–19.

47.

Mirjalili

, Mirjalili

S.M.

and Lewis

, Grey wolf optimizer, Advances in Engineering Software. 69 (2014), 46–61.

48.

Askari

, Saeed

and Younas

, Heap-based optimizer inspired bycorporate rank hierarchy for global optimization, Expert Systems with Applications 161 (2020), 113702.

49.

Şenel

F. Ahmet

, Gökçe

, Yüksel

A.S.

and Yiğit

, A novel hybrid pso–gwo algorithm for optimization problems, Engineering with Computers 35(4) (2019), 1359–1373.

50.

Ghatak

, Rup

, Majhi

and Swamy

M.N.S.

, Hsajaya: An improved optimization scheme for consumer surveillance video synopsis generation, IEEE Transactions on Consumer Electronics 66(2) (2020),144–152.

51.

Sun

, Xing

, Ai

and Lao

, A tracking based fast online complete video synopsis approach. In Proceedings of the 21stInternational Conference on Pattern Recognition (ICPR2012), IEEE (2012), 1956–1959.