Stacked collaborative transformer network with contrastive learning for video moment localization

Abstract

Video grounding intends to perform temporal localization in multimedia information retrieval. The temporal bounds of the target video span are determined for the given input query. A novel interactive multi-head self-attention (IMSA) transformer is proposed to localize an unseen moment in the untrimmed video for the given image. A new semantic-trained self-supervised approach is considered in this paper to perform cross-domain learning to match the image query – video segment. It normalizes the convolution function enabling efficient correlation and collecting of semantically related video segments across time based on the image query. A double hostile Contrastive learning with Gaussian distribution parameters method is advanced to learn the representations of video. The proposed approach performs dynamically on various video components to achieve exact semantic synchronization and localization among queries and video. In the proposed approach, the IMSA model localizes frames greatly compared to other approaches. Experiments on benchmark datasets show that the proposed model can significantly increase temporal grounding accuracy. The moment occurrence is identified in the video with a start and end boundary ascertains an average recall of 86.45% and a mAP of 59.3%.

Keywords

Contrastive learning gaussian parameter self-attention transformer temporal localization video grounding

1. Introduction

The fast growth of video data necessitates the development of effective indexing and retrieving services. Video grounding can be applied for many applications such as video surveillance [5], video retrieval systems, sports video analysis, automated driver systems, etc. Many researchers are focussing the video domain on surveillance systems. With the increasing growth of video data, analyzing compressed video content typically begins by dividing the video into multiple related frames known as Groups of pictures (GOP), from which one or more keyframes or indicative frames can be recovered for each GOP. Broadcasters broadcast the highlights of the sports event at different time intervals. These highlight events are detected and clustered [20] based on the representative clips concerning skimming time. The similarity is measured between the query and video clips for retrieval. A distance metric such as hamming distance, Euclidean distances, and zero normalized correlation coefficient (ZNCC) is computed to measure the similarity among the frames consisting of images. Contrastive loss, triplet loss, and quadruplet loss are computed to train the similarity learning model. This method focuses on pairs, ensuring that the distance between an anchor and a positive sample is closer than to a negative one by a set margin. It also brings samples of the same class closer to their class centroids in the embedding space, improving representation of similarities and differences. There exists a lack of temporal consistency among the successive frames of the video.

The complexity of modern deep learning algorithms has increased greatly, allowing them to extract characteristics and make significant connections; however, this comes at the expense of increased time and resource requirements. In contrast to algorithms used for picture classification and object recognition, which may do multiple tasks simultaneously, the tracking method does not. It’s a multi-purpose algorithm that can find things (object detection), put them in categories (localize), and maintain tabs on them (classification). This method is quite difficult to train theoretically and requires a considerable amount of effort. Additionally, it is important to remember that the tracking algorithms’ speed during inference time is a critical factor in the reliability of the data they provide. For real-time object tracking models, boosting tracking performance is very crucial.

Many deep learning techniques such as Convolutional neural network (CNN), and feed-forward neural network (FFN), are used to extract the features from the given image and video. These networks train themselves to learn the features of image or video data without labels which are known as self-supervised learning models. Video localization includes two major problems such as essential representative frame selection and selecting the frames consisting of maximum semantic values. This work concentrates on video moment localization to address these key issues. To retrieve the video and localize a particular frame from a large-scale database, existing pooling methods have a high computational cost. The temporal localization of frames based on image query with high accuracy is a challenging task. Transformer network models are used to convert the extracted features into their context vector representatives with the use of encoders and decoders. Recurrent neural networks (RNN) such as Long-Short Term Memory (LSTM) are preferred to process and do predictions with sequential data. Attention mechanism (Guo et al., 2021) is introduced in the transformer to implement cross learning model from the outcomes of deep neural network (DNN) models. Attention is done in the encoder-decoder layers [16, 32] to concentrate more on visual features and their embeddings.

Contrastive learning (CL) is another approach to self-supervised learning which is being emerged as an additional efficient method for trained deep neural networks. Unlike generative models, CL is a discriminative method that tries to group similar data and varied ones apart. Figure 1 shows the self-supervised learning pipeline to perform contrastive learning. A model to self-train itself first augments the input data. One instance from the training sample is chosen and a changed variant of the data is obtained using adequate data augmentation techniques. Then the augmented data is passed to the encoder and from those encoded vectors, a pair of images are chosen (positive or negative) to compare the similarity of the embeddings. A similarity metric is being used to compare the similarity of two embeddings. A contrastive loss is estimated based on the feature representations of the pictures recovered from an encoder network, especially for computer vision tasks.

Figure 1.

Self-supervised training pipeline.

Video retrieval tasks are performed for the given text query or a clip of video. Many works use holistic approaches such as Euclidean distance [2], similarity estimation, and ranking scores. These scores are used to map the embedding space of video and text. Retrieving corpora of video moment is the extension of single moment retrieval in the video. LSTM is used to transfer the video spatial information into flat representation. But there exists a loss in spatial information [15, 33]. Hence, a spatial encoder or attention mechanism is needed to obtain the spatial relationship between video and query. A spatial attention mechanism is to be created to learn frame regions to pay attention to each word in the query.

In the proposed research work, for a given image query, the semantically related occurrence of the query image is detected in the given video and the moment sequence-sensitive localizer is framed to localize the contender video moment for retrieval. A multi-head self-attention transformer model is developed to compute the attention score of visual words in the video. The self-supervised contrastive learning is used in the proposed work to localize the video moment for the given input query on computing the similarity as well as contrastive loss.

The major contributions are: –

An Image-based moment Localization is proposed which uses an image query to locate unnoticed behaviours in full-length rough videos.

–

Proposed a moment localization framework that uses interactive multi-head Self Attention (IMSA) mechanism with contrastive learning to develop the temporal progression of events based on input query.

–

Intercept and transform the active spatiotemporal features of video and learn rich relational features. IMSA network outperforms convolutional networks.

2. Related works

Many video grounding works are rank-based or attention-based methods to localize the target video extent on computing the finest matching score. Video grounding performs action recognition or concept localization. It identifies all the action classes that occurs in the video sequence. Action recognition is more specific that falls under multi-class classification problem. An action may occur from frame A to frame B and the motivation is to determine the boundaries of those action occurrence. Reinforcement learning is also deployed to filter the video moment with temporal bounds.

Mallick et al. [18] detected the salient region and extracted the salient features of the extracted keyframes. Matching is performed between the query video features and salient features of the extracted key frames which are stored in repositories. Araujo and Girod [1] proposed an asymmetric approach to video retrieval using image queries. The video is segmented, and fisher vectors and embeddings are constructed with hashing based on the bloom filter. Then the image descriptors and video descriptors are compared to retrieve similar frames. Videos are represented using fisher vectors and bloom filters and are temporally aggregated for retrieval purposes. Optimization of fisher vectors needs improvement while performing an asymmetric comparison. Peng and Ngo [20] implemented clip-based similarity measurement for video retrieval. The irrelevant frames are filtered out, and similarity matching (consisting same visual information) is done between the query clip and video. On maximizing the similarity, the clustering of frames is performed and retrieved. The computation complexity is based on the number of shots and edges that match. Rahmani and Zargari [21] introduced a prediction unit size feature vector on B and P frames. Using motion vectors and color histogram, retrieval is performed, and ranking is done on the retrieved video. Lin et al. (2020) extracted the features of video using deep CNN and retrieved the videos based on the preserving similarity of the spatiotemporal relationship in pooling frames. Adam optimizer has been used to update the parameters. The rank-based structured loss function is computed with L2 normalization and Hamming distance, mean average precision is the evaluation metric used.

2.1 Sentence query-based video retrieval

Shen et al. [22] performed temporal hashing along with the gated recurrent units of 10 layers to retrieve the video on estimating hamming ranking criteria for searching. Yu et al. [27] implemented a multilinear pooling method to perform video grounding for the given sentence query. The text and video features are integrated with multitask learning and losses were regularized. Wang et al. [24] developed an interactive transformer to model the text query and video pair interactions and predict the temporal boundaries in the video. Yang et al. [25] developed a local correspondence network to video moments by hierarchically representing features and modelling the target moments by relating the correspondence of local features. This models GPU memory usage. Gao et al. (2020) performed video knowledge transfer among graph-based neural networks and the embedding space to retrieve the moments in a video for the given sentence. Zhang et al. [30] provided an excellent survey on temporal grounding tasks on a video.

2.2 Action recognition

Gao et al. [7] proposed a cross model for temporal activity localization from an untrimmed video based on the query. Temporal regression has been concentrated on computing the alignment scores and clip regressions. Jaiswal et al. [12] have provided an extensive survey on contrastive learning for self-supervised learning models. Zhang et al. [29] have implemented contrastive learning along with the unimodal encoding method to retrieve the queried moment from the video. Interactions between text and visual features are performed and fine-grained video retrieval is done. Chen et al. [4]implemented a multimodal network for temporal grounding. Table 1 shows the previous works of contrastive learning with their pros and cons.

Table 1
Contrastive learning.

Method Objective Pros and cons

MoCo (Moment Contrast) [11] FIFO Queue-based dynamic dictionary is used to store the encoded representations with a list of keys. The similarity between query and momentum encoder keys is computed. Pros: Decouples batch size.
Cons: Not differentiable.
Does not rely on backpropagation to update the key encoder.

Barlow Twins [2] Minimize the redundancy among the vectors of distorted samples and create a cross-correlation matrix. Pros: Avoids trivial constants. Robust to varying batch sizes during training.

SimCLR [28] Learn the visual representation by maximizing the agreement among augmented views of the same feature. Pros: Provides efficient MLP projection head and strong augmentation.
Cons: It requires a large batch size to have enough negative samples. Thus, for reduced bath size, performance drops.

BYOL (Bootstrap Your Own Latent) [9] Learn the representation without using negative samples. Relies on pseudo labels. Cons: On removing batch normalization, performance is less.

Method	Objective	Pros and cons
MoCo (Moment Contrast) [11]	FIFO Queue-based dynamic dictionary is used to store the encoded representations with a list of keys. The similarity between query and momentum encoder keys is computed.	Pros: Decouples batch size. Cons: Not differentiable. Does not rely on backpropagation to update the key encoder.
Barlow Twins [2]	Minimize the redundancy among the vectors of distorted samples and create a cross-correlation matrix.	Pros: Avoids trivial constants. Robust to varying batch sizes during training.
SimCLR [28]	Learn the visual representation by maximizing the agreement among augmented views of the same feature.	Pros: Provides efficient MLP projection head and strong augmentation. Cons: It requires a large batch size to have enough negative samples. Thus, for reduced bath size, performance drops.
BYOL (Bootstrap Your Own Latent) [9]	Learn the representation without using negative samples. Relies on pseudo labels.	Cons: On removing batch normalization, performance is less.

2.3 Attention for understanding video

Video understanding involves variants such as convolution, self-attention, motion learning, etc. Convolution and attention aim to understand the video by capturing its temporal dynamics throughout several frames [6]. Spatiotemporal convolution is a 3D convolutional model attached with self-attention for better video understanding and semantic-based video retrieval tasks. Though vision transformers are designed for this purpose, it lacks in relating the inter and intra features of video (spatial and temporal) to context vector. A video with input features (height, width, time) is transformed into a context vector using neighbour feature information. The transform function is applied on each position of the features X with a learnable weight W and is being mapped to a context vector Y. Convolution is the operation of translating the features to an equivariant form by applying kernels. The region of interest and its features are extracted using a convolution neural network with many kernels, but convolution persists stable. This does not affect the target visual but due to channel weight dependency, redundancy occurs, and complexity increases.

Self-attention is a dynamic transform mechanism consisting of encoder and decoders, that generates a context vector attention map using a target. It then accumulates the context using a dynamic kernel attention map. The attention mechanism has three computations such as query, key, and value for the given input and target. Using the learnable embeddings of Q, K, V matrices, the attention embeds the target into a query and then projects the context into key and value matrices. Attention map computation involves a scaled dot product of query and key matrices. Here the content attention is done with the positional embedding information. The self-attention [23, 14] transformer aggregates the context embedding values using the attention map as kernel weights and thus an interaction takes place between the context and the position. A SoftMax activation function is applied to the aggregated context vectors based on the target. It is flexible and consumes only a few parameters. SoftMax reduces the computational complexity and hence self-attention is used for video understanding, classification, and retrieval. Existing transforms lack providing context-to-context interactions and content information to position vector interactions. Thus, video understanding, and localizing a moment in the video is a crucial task as video consists of spatial and temporal contents.

3. Proposed methodology

An overview of the proposed Spatio-temporal multi-head attention moment Localization Model is illustrated in Figure 2. Initially, the model encodes the image and video separately with an attention mechanism. A set of untrimmed videos of any length consisting of a sequence of frames with optional subtitles is given as input to the proposed model. Each video is correlated with the image query that corresponds to a video segment representing or containing semantically similar image information. For the given image query Q, the main task is to retrieve the temporal moment with a boundary of starting frame position and ending position of video frames. A unimodal approach is proposed to perform efficient video moment localization and retrieval with high accuracy. Unimodal techniques such as temporal segment networks and temporal action proposals analyze video data only, making them simpler but potentially less effective than multimodal approaches, which incorporate text, image, and video data for richer context. The position and input embedding are passed to the IMSA model explained in the following section. During the process, an interactive multi-head self-attention (IMSA) transformer model is simulated to determine the attention score and alignment score among the encoded features. The interactive multi-head self-attention transformer is a model designed for detailed retrieval by analyzing video content and query picture regions. It employs multiple attention mechanisms to pinpoint relevant information in both the video and query image, facilitating accurate matching and retrieval of pertinent content. The features are then normalized. Also, contrastive learning (CL) with contrastive loss estimation is computed at the moment sequence-sensitive localizer which provides better accuracy. Further, to enhance the performance, Gaussian Mixture Model (GMM) parameters are trained for localization among the representations of latent space. Parameter optimization in Gaussian Mixture Models (GMMs) for latent space representations aims to refine parameters such as means and covariances iteratively. This enhances the model’s ability to accurately capture data distribution nuances, improving localization of data points within the latent space.

Figure 2.

Interactive Multi-head Self Attention (IMSA) for moment localization.

Each video V consists of many clips denoted as {Ci} of fixed lengths. Clip-level representations are used to learn semantic alignments which may consist of annotations for the candidate moments within the joint embedding space. Representing each clip individually reduces computational complexity, allowing focused analysis. Semantic alignments align clips with concepts, aiding accurate recognition and interpretation, improving task effectiveness in surveillance, action recognition, and content understanding. Each query image consists of many regions of interest (ROI). In the first stage of the model, faster CNN is employed to extract the features. Further, in the second stage, the localization of ROI is fine-tuned. C3D, a 3-dimensional convolutional neural network is used to extract the high-level features of the video. Similarly, the features of the given query image are also extracted using a faster convolution neural network.

Based on those regions having high precision, self-attention is performed on the video to detect semantically and localize the particular temporal moment for a certain window length. 4096-dimensional visual features are extracted from the fully connected layer and are convolved to 512-dimension using hyperbolic tangent functions with a weight parameter and, the visual embeddings are obtained. Positional encoding vectors and input embeddings of the video features are passed to the multi-head self-attention encoder transformer which consists of stacked conventional N layers. Extracted features are being projected to latent dimension d with weight and bias. A convolutional stacked attention-based transformer is used to project the moment sequence representations and the query representations in joint embedding space. CSAT combines convolutional layers to extract features from input sequences, followed by stacked attention mechanisms to emphasize important segments. Transformers are used to capture relationships between input parts, aiding in tasks like sequence understanding and prediction. Thus, an interactive multi-head self-attention transformer is employed to explore the relationship of those query image regions in the video for fine-grained retrieval.

3.1 IMSA attention-based transformer

The goal of the IMSA localizer is to predict the boundaries of unseen activities in the video containing contender-relevant temporal moments during the inference process. Multiple objects of the query are tracked in the video based on the object of interest. Each layer has an encoder, followed by layer normalization, a feed-forward network with normalization. Here a stack of three layers is used in the transformer block. The learning target for the queries is to explicitly focus on differentiating intra-video moments from inter-video global semantics. The algorithm draws a box around the object and uses this to determine its kind and location. Giving each thing its own identifier (ID). Continuously focusing on the detected object as it travels between frames and saving any pertinent data. Additive margin-based triplet loss and maximum margin-based triplet loss are used to train the model in two different performance improvements. Maximal margin-based triplet loss maximizes the margin between anchor-positive and anchor-negative pairs, promoting distinct embeddings for different classes. Additive margin-based triplet loss adjusts the margin dynamically based on similarity, potentially improving generalization. Algorithm 1 illustrates the steps involved in multi-attention for temporal localization. The localization with the most similar representation is retrieved from the video sequences during the inference stage for the given query.

4. Contrastive learning for Spatio-temporal Localization

Contrastive learning is used to learn the visual representations and features extracted from the encoders. It is used to maximize the mutual information among input data and visual representations. A small feed-forward neural network is used to map the encoders’ output representations to 128 dimensions of latent space. Contrastive loss and cross-entropy loss are calculated to determine the boundaries of the target moment in a video.

Batches of a specific size, N, are created from raw image and video to begin contrastive learning at the frame level. A stochastic IMSA transformation function is performed on each image in this batch to create a pair of two images. To obtain visual features, each enhanced image in a pair is sent through an encoder. After that, the representations of the two enriched images are sent via a non-linear dense layer, a ReLU, and yet another dense layer. Feature extraction and transformation play a crucial role in enhancing a model’s capability to recognize complex patterns and relationships within data. This process significantly boosts performance in tasks like image classification, object detection, and image segmentation. These images are projected into a representation after being passed through a sequence of these layers to apply non-linear encoding processing.

Embedding vector is obtained for every augmented image in the batch. Cosine similarity is used to determine similarity between two augmented representations of an image. In augmented image representations, features are typically depicted as vectors. Comparing these vectors using cosine similarity measures their directional alignment. Higher cosine values imply greater feature similarity, useful for tasks like image retrieval and classification. Then the contrastive loss is computed using the loss function. For each augmented pair of visual words of query and frame, apply the SoftMax function to estimate the probability based on similarity.

The encoder and projection head approximations gradually improve as a result of the loss, and the representations acquired bring similar visuals closer together in space. Contrastive learning outperformed earlier self-supervised approaches on ImageNet, according to the results.

4.1 Moment localization with contrastive learning

The encoded features of the given query $f_{q}$ and video features $F_{V}$ are further encoded with the feed-forward network (FFN) layer with weight $w_{q}$ and bias $b_{q}$ as shown in Eq. (1). At each time step, the agent observes the current state, which consists of a bag of visual words and a candidate clip of the video. Typically, different sorts of annotations are applied to candidate moments inside a point embedding space: temporal label, semantic labels, and relevance scores.

\begin{aligned} f_{q}^{^{'}} = w_{q} \cdot f_{q} + b_{q} \end{aligned}

(1)

The clip is defined by a bounding window where the start and end are frame numbers. The similarity score between the given image query $q$ and the video $V$ consisting of clips ${c_{1} \dots c_{i}}$ are computed. Similarity score $S_{v q}$ is estimated as:

\begin{aligned} S_{v q} = F_{V}^{T} \cdot f_{q}^{^{'}} \in R^{n} \end{aligned}

(2)

Where $v$ consists of clips and sentences is optional.

The starting score, $S_{s t}$ and ending score, $S_{e d}$ of the target moment is created by using the Conv1D [11]:

\begin{aligned} S_{s t} = Conv 1 D_{s t} (S_{v q}); S_{e d} = Conv 1 D_{e d} (S_{v q}) \end{aligned}

(3)

As layer normalization is performed, the feature vectors are being gaussian distributed. The probability distribution of the boundary with starting $P_{s t}$ and ending position $P_{e d}$ is computed using SoftMax function as:

\begin{aligned} P_{s t / e d} = s o f t m a x (S_{s t / e d}) \end{aligned}

(4)

The target moment boundaries are derived by maximizing the combined probabilities. Thus, the best boundaries are detected with the maximized scores. $L_{1}$ Loss value is computed using the cross-entropy function, to fix the best matching boundaries. Cross-entropy will compute a score that highlights the average difference between the observed and predicted probability distributions. It assumes one hot encodings $h_{s t}$ and $h_{e d}$ .

\begin{aligned} L_{1} = \frac{1}{2} \times (f_{C E} (P_{s t}, h_{s t}) + f_{C E} (P_{e d}, h_{e d})) \end{aligned}

(5)

Where, $f_{C E} (.)$ is the cross-entropy loss function.

4.2 Similarity computation for video moment retrieval

The query is encoded into vectors $f_{q}$ and the video is encoded as $F_{V}$ with its features of visual appearance and subtitles if available. Once the video is encoded, it consists of a sequence of vectors each representing smaller segment of a video. The encoded video is represented as $F_{V} = {F_{V}^{1}, \dots F_{V}^{n}}$ .

The similarity between the query and the video sequence elements is estimated using cosine similarities on the features extracted. The features include text features like TF-IDF vectors or embeddings from the query, visual features from video frames such as deep learning-based embeddings, and temporal features like LSTM or Transformer-based embeddings to capture temporal dynamics in the sequence. Cosine similarity is used to measure the cosine angle between two projected vectors. If the angle is smaller between these vectors, then the similarity is higher. It is calculated by finding the dot product of two vectors and dividing it by the product of its magnitude. Then cross-correlation is computed.

\begin{aligned} S i m (f_{q}^{i}, F_{V}^{j}) & = cosine similarity (f_{q}^{i}, F_{V}^{j}) \end{aligned}

(6)

\begin{aligned} {S i m}_{m q} & = n o r m (F_{V}^{T}) n o r m (f_{m q}) \end{aligned}

(7)

Here norm (.) is the L2 normalization. The similarity score is a scalar value. The maximum similarity score is considered for representing the highest match amongst query and video.

During the training of the video retrieval objective, a triplet margin loss function is adopted. The accurate identification of boundaries is achieved by minimizing loss functions, such as cross-entropy or mean squared error, which measure the deviation from ground truth labels. Through iterative optimization, the model learns to pinpoint the precise boundaries of the target instant. This function is highly used for content-based retrieval tasks. This $L_{2}$ , triplet margin loss function estimates the relative similarity that exists between the query and video clips and then computes the loss based on the three input tensors $(a, p, n)$ and also margin value $>$ 0. The tensor $a$ represents the anchor value, $(p, n)$ indicates the positive and negative values respectively. For each video segment in the mini-batch, $L_{2}$ is expressed as,

\begin{aligned} L_{2} (a, p, n) = max {0, d (a_{i}, p_{i}) - d (a_{i}, n_{i}) + Δ} \end{aligned}

(8)

Where $Δ$ is the margin value, a learning parameter set as 0.1 and $d (.)$ is the distance metric. the delta distances between anchor-positive and anchor-negative to be larger than the margin. The distance is measured as,

\begin{aligned} d (x_{i}, y_{i}) = | | x_{i} - y_{i} | |_{2} \end{aligned}

(9)

The IMSA model is trained based on the similarity relationships known as contrastive loss. The embeddings are learned with contrastive loss. This loss function computes the similarity among objects using similarity functions. This loss function is used to optimize the nonlinear parameters. Combining these two losses, minimize the loss and learn to approximate the boundaries in the embedding space.

The time complexity and space complexity on training IMSA model are shown in Table 2. Here Batch size (S), for the given input size (N), context size (L), channel dimension (D), latent channel dimension and number of queries (Q) are used to train an IMSA network.

Table 2

Complexity measure.

Method	Time complexity	Space complexity
Self-attention	$O ({S N L}^{2} D)$	$O (S N L (L + D) + L^{2} D)$
Enhanced IMSA	$O ({S N L D}^{2})$	$O ({S N D}^{2} + L D^{2})$
IMSA with many queries (Q)	$O ({S N L D}^{2} / Q^{2})$	$O ({S N D}^{2} / Q^{2} + S N D + L D^{2} / Q^{2})$

4.3 Frame-wise contrastive learning

The video retrieval function anticipates that the video has a similar moment to the query, hence frame-wise contrastive learning emphasizes moment specificity inside a particular pair of the video-image query. Video retrieval functions are crucial for quickly finding and accessing specific video content. They enable users to search within videos based on keywords, topics, or even visual elements, enhancing the efficiency of video content management and enabling more precise content discovery. The visual features that are within the target moment’s boundary are considered forefront or positive samples, while others are considered background or negative samples. Addressing class imbalance in datasets involves various strategies such as oversampling the minority class, augmenting its samples, utilizing robust algorithms like Random Forest or AdaBoost, adjusting misclassification costs, and employing techniques like SMOTE for generating synthetic samples. The contrastive loss is then determined by calculating mutual information among both the query and the optimistic video features. Contrastive loss distinguishes images based on their similarity. A similarity score is calculated by comparing the feature or latent layer to the target using a similarity metric and training with the target. When both inputs are the same (positive pairs), the target is 0. Collaborating contrastive learning with Gaussian Mixture Model-Expectation Maximization (GMM-EM) is performed to maximize the likelihood to localize the video grounding process.

4.4 GMM parameters estimation

For recognizing moving objects, the Gaussian Mixture Model (GMM) is a well-known ambient modelling technique. Usually, GMM is used to cluster the data represented as a posterior probability distribution. All data points are created using a composite of bounded Gaussian distributions having uncertain parameters in this method. When dealing with uncertain parameters, like mean and variance, common methods include maximum likelihood estimation and Bayesian inference. These approaches account for constraints and provide estimates that align with the data, making bounded Gaussians a good choice for modeling such data points. The GMM technique creates models for each pixel using numerous flexible Gaussian distributions and updates the model using an online approximation. The threshold (T) and learning rate ( $γ$ ) are two parameters required by the approach.

You et al. [26] have introduced the hard GMM to the self-attention transformer to improve the quality of cross attention. But this hard-coded attention performance was less compared to the base transformer. Zhou et al. [31] captured the correlation among features using the likelihood of expectation-maximization algorithm involving GMM parameters and thus features are discriminated for the recognition task. Gaussian kernels control the mapped features sequential range.

GMM squeezes the probable outcomes into smaller regions. This indicates that it is not only limited to discrete but also continuous data. GMM is the weighted sum of Gaussian distributions of multivariate (K). It is expressed as:

\begin{aligned} p (\hat{x} | Θ) = \sum_{k = 1}^{K} w_{k} \cdot N (\hat{x}; μ_{k}, \sum_{k}) \end{aligned}

(10)

Where $\hat{x}$ is the encoded representation, $Θ = {w_{k}, μ_{k}, \sum_{k}}$ , $w_{k}$ represents weights, $μ_{k}$ , $\sum_{k}$ represents mean and covariance respectively. To compute the parameters of GMM, Expectation-Maximization is used. The loglikelihood score is computed as $\log (p (\hat{x} | Θ))$ . $N (\hat{x}; μ_{k}, \sum_{k})$ is the probability density function of encoded feature vectors of n-dimension in a latent space obeying Gaussian distribution and it is defined as:

\begin{aligned} N (\hat{x}; μ_{k}, \sum_{k}) = \frac{\exp (- \frac{1}{2} {(\hat{x} - μ)}^{T} \sum_{k}^{- 1} (\hat{x} - μ))}{\sqrt{| 2 π \sum_{k} |}} \end{aligned}

(11)

The parameters used in the algorithms are shown in Table 3.

Table 3

Parameters used.

Symbol	Description
Vi	Video
Qi	Query Image
$W_{Q}$	Query weight
$W_{K}$	Key weight
$W_{V}$	Value weight
$q_{i}$	Query vector
$k_{i}$	Key vector
$v_{i}$	Value vector
$a_{i j}$	Attention weight
$d_{k}$	Dimension of the key vector
$η$	Number of attention head
$d (.)$	Distance metric
$Δ$	Margin value
$γ$	Learning rate
$L$	Contrastive loss
$L_{1}$	Cross-entropy Loss
$L_{2}$	Triplet margin loss
K	Multivariate Gaussian distributions
$\hat{x}$	Encoded representation
$μ_{k}$	Mean
$\sum_{k}$	Covariance
$\log (p (\hat{x} \| Θ)) .$	Loglikelihood score
$N (.)$	Probability density function
R	Recall
mIoU	Mean IoU
[ $G_{s}, G_{e}$ ]	Ground truth clip for query Q

Likelihood regularization is related to the probability distribution. Segment features and embeddings are taken to predict combined segment-embedding relevance likelihood. Neighbourhood information is incorporated to give a smoothing effect on the outputs of the prediction probabilities. This helps to perform intercession in video grounding tasks with high accurate temporal boundaries inference. This is known as regression loss, which fine-tunes the distance among predicted moment chunks and ground truth ones moved from its boundaries. Contender generator eliminates the video chunks which are not relevant to the class. It also reduces the searching space using high recall transformer model. This improves the performance of likelihood computation and high precision for top K results.

5. Experimental analysis

5.1 Dataset

Experimental analysis is conducted on three different benchmarks datasets such as SumMe, Activity Captions, and charades-STA designed to localize the moments in the content-based video retrieval. Jie et al. (2020) discussed a large-scale dataset to perform moment retrieval tasks from videos. Activity Net captions is the dataset with temporal annotations [19] is used for object tracking. Video in this dataset consists of many segments with labels characterizing the actions. The Charades-STA dataset were often used for video captioning in the retrieval of video moments. The details of the dataset are mentioned in Table 4.

Table 4
Dataset description.

Dataset Activity Net captions Charades-STA SumMe

Number of videos 14950 8354 25

Resolution 480 $\times$ 270 1280 $\times$ 720

Average Video length 42 s 30 s 6 min

Frames per Second (FPS) 30 FPS 30FPS 30 FPS

Activity Outdoor activity Indoor activities Outdoor activity

Dataset	Activity Net captions	Charades-STA	SumMe
Number of videos	14950	8354	25
Resolution		480 $\times$ 270	1280 $\times$ 720
Average Video length	42 s	30 s	6 min
Frames per Second (FPS)	30 FPS	30FPS	30 FPS
Activity	Outdoor activity	Indoor activities	Outdoor activity

5.2 Implementation details

Our model is implemented in Ubuntu 16.04, YOLOv3, Tensorflow 1.8.0, with CUDA 11.0 and cudnn 8.0.5. All experiments are conducted on a workstation with dual NVIDIA GeForce GTX 3070 GPU. The network is trained with 50 epochs and with a minimum batch size of 30, the learning rate of 0.001. Moment and weighting factors are introduced, and the learning rate is reduced for every single epoch. For the inputted query image, 4096-d features of several regions are extracted with its position vector embedding using C3D CNN. 3D convolutional network is used to locate the boundary of the actions recognized within the video. Its global features dimension is reduced to 512 without any redundancy. Similarly, the visual features of 4096-d are extracted from video frames using C3D CNN for each unit consisting of 16 frames. Feature mapping is performed with a kernel of sizes 3 and 5 applied in convolution layers. On passing these features to the conventional stacked IMSA attention model, the feature of 256-d is projected. Layer normalization is added with the activation function. In contrastive learning, horizontal flipping and random cropping are done for data augmentation of the query image. The gaussian distribution data with a probability of 0.5 is used for training purposes. To reduce the combined loss, a Stochastic descent gradient optimizer is used. The proposed model is trained until the loss gets smoothened. Figure 3 shows the moments being localized in the videos for the given image query to the IMSA network. After empirically analyzing with various parameters, the efficient result is achieved for the above-mentioned metrics.

Figure 3.

Moments localized in the video at right for the given image queries at left: (a) Pouring water (b) Girls playing tennis (c) Air force videos (d) Cooking pasta (e) Sea riding.

5.3 Performance metrics

Video moment retrieval is measured by the Recall metric (R). The performance of the proposed model is evaluated based on two criteria such as mIoU (mean IoU) and IoU@R. The Intersection over Union (IoU) is estimated for all pairs of video segments and the ground truth values. The projected bounding box’s overlap with the actual bounding box is represented by an Intersection over Union. If the IoU is high, then the projected bounding box coordinates are very similar to the ground truth box coordinates. Then the mean of IoU for all made predictions is computed. Figure 4 represents the Recall and IoU curve for the proposed IMSA model compared to GTAN and SCDM approaches validated on ActivityNetCaptions dataset. To discriminate moments from distinct videos, the proposed learning target use global semantics. To provide it, the network is trained to align corresponding video-query pairings in the subspace, where video-query relevance R is estimated based on the moment similarity. Figure 5 illustrates the Recall vs IoU curve of the proposed IMSA model validated on Charades-STA dataset. As a result, the video-query relevance score is utilized to evaluate and report on the model’s effectiveness in identifying or retrieving the proper video given an image query.

Figure 4.

Recall-IoU curve on ActivityNet_Caption.

Figure 5.

Recall-IoU curve on Charades-STA.

The evaluation criteria used to measure the performance of the video retrieval task is R@K for three different datasets such as Activity Net captions, charades-STA, and sum me dataset. R@K value indicates the percentage of correctly predicted moments for the input query which is found in top K retrieved videos. IoU is calculated for the predicted video chunks and ground truth chunks of video for the query image. The percentage of predicted moments for which IoU is greater than R is computed and denoted as IoU@R. In previous methods, IoU scores are estimated for top k reimbursed video clips, and sliding windows are aligned based on the confidence score of those returned moments. Here IMSA model along with contrastive loss and GMM loss hyperparameters, the model hunts down the video until the best aligned moment is found in that clip and returns it. The IoU scores are calculated for R@1 and R@5 for two different datasets and are shown in Tables 5 and 6. The IOU is calculated between the clip of the state at time t, [ $W_{s}^{t}, W_{e}^{t}$ ], and the ground truth clip for query Q, [ $G_{s}, G_{e}$ ] as follows:

\begin{aligned} I o U = \frac{min (W_{e}^{t}, G_{e}) - max (W_{s}^{t}, G_{s})}{max (W_{e}^{t}, G_{e}) - min (W_{s}^{t}, G_{s})} \end{aligned}

(12)

Table 5

Performance comparison of ActivityNetCaptions.

	R@K, IoU $K =$ 1			R@K, IoU $K =$ 5
Method	IoU $=$ 0.3	IoU $=$ 0.5	IoU $=$ 0.7	IoU $=$ 0.3	IoU $=$ 0.5	IoU $=$ 0.7
DRN	–	44.07	23.86	–	76.52	50.41
CBP	54.41	35.67	17.79	77.35	65.02	41.68
FVMR	60.36	44.8	26.78	86.07	77.04	61.11
TAN	59.34	44.87	26.02	85.61	77.24	61.24
SCDM	55.04	36.84	19.94	77.36	65.05	41.62
IMSA	60.24	45.52	27.34	86.45	76.87	61.97

Table 6

Performance comparison of Charades-STA.

		R@K, IoU $K =$ 1		R@K, IoU $K =$ 5
Method	Features	IoU $=$ 0.5	IoU $=$ 0.7	IoU $=$ 0.5	IoU $=$ 0.7
DRN	I3D	52.98	31.69	89.02	60.04
SCDM	I3D	54.38	33.36	73.85	57.96
IMSA	I3D	55.02	33.25	89.27	57.24
FVMR	VGG	42.41	23.86	84.03	50.09
MAN	VGG	41.01	22.79	83.86	51.06
CTRL	VGG	23.64	08.88	58.93	29.48
IMSA	VGG	42.36	24.21	84.25	50.25
CBP	C3D	36.82	18.92	71.02	50.20
ACL	C3D	30.50	12.31	64.76	35.26
IMSA	C3D	37.16	18.22	80.23	48.84

6. Results and discussion

The variants of our system are tested for IoU $=$ {0.3, 0.5, 0.7} and Recall@K $=$ {1, 5}. Here K is a randomly chosen sliding window with IoU $=$ m. All compared methods use the same features of I3D, C3D, and VGG. DRN, CBP, and SCDM are anchor-based methods for temporal grounding that involves object detection also. CTRL and MAN are sliding window approach networks. All models are evaluated on mean average precision (mAP) by retrieving top 100K candidate segments for each class. Some of these methods such as SCDM, and CTRL uses Bi-LSTM for encoding the sentence query. Few methods use text extractors. Visual features are extracted along with input embedding and positional encoding. It is then further passed to the feed-forward network to compute the attention score and distribution probabilities. Hyperparameters with contrastive learning in the network help the moment localizer to lose ground in the accurate boundaries for the query. An ablation study on video moment retrieval indicates that, on including GMM parameters in the multi-head attention transformer, the performance of the target moment localization model degraded. Expectation (E-step) and Maximization (M step) are two basic steps of the GMM model. Increasing the iterations of Expectation (E-step) and Maximization (M step) of EM algorithms, the complexity of the model increases. Hence the search space is reduced in contender generation and the IMSA transformer model is fine-tuned using high precision.

The Average Recall on different IoU of the proposed IMSA transformer model achieves 86.45% and 80.23% on ActivityNet and Charades-STA respectively. The mean average precision (mAP) is determined by first determining the Average Precision (AP) for each class, and then taking the mean of this value across all classes. The mAP takes into account the cost of false positives (FP) and false negatives (FN) and reflects the trade-off between precision and recall (FN). Evaluation of an object recognition and tracking model’s precision can be performed with the help of the mAP. On tracking the objects on processing videos at 30FPS has a mAP of 59.3% on ActivityNetCaptions dataset. Figure 6 illustrates the Average recall vs Average number of proposals curve obtained by the proposed IMSA model. The computational cost is reduced by using multiple queries ad the performance measure illustrating number of parameters, FLOPs and top-1 and top-5 results accuracy is shown in Table 7.

Table 7
Performance measures.

No. of queries FLOPs Parameters Top-1 Top-5

1 248.6 G 27.8 M 50.2 78.5

2 89.7 G 23.7 M 51.4 79.4

4 51.4 G 22.3 M 51.7 79.9

6 45.6 G 21.7 M 53.4 89.2

8 33.8 G 20.7 M 55.1 80.6

10 31.6 G 20.3 M 52.3 80.2

No. of queries	FLOPs	Parameters	Top-1	Top-5
1	248.6 G	27.8 M	50.2	78.5
2	89.7 G	23.7 M	51.4	79.4
4	51.4 G	22.3 M	51.7	79.9
6	45.6 G	21.7 M	53.4	89.2
8	33.8 G	20.7 M	55.1	80.6
10	31.6 G	20.3 M	52.3	80.2

Figure 6.

Average recall-average proposals curve.

The results reveals that the temporal moment localization by IMSA is efficient compared to other approaches. Processing the dark or light frames in the video is a challenging task for the proposed model as the misclassification rate is more. Thus, set of features may be able to retrieve frames having range of elimination variations may fail to retrieve noisy frames; while a set of features may effectively retrieve frames with various elimination variations, it might struggle with noisy or irrelevant information. This underscores the challenge of dealing with noisy data in video moment localization tasks. It extracts only video frames and fails to detect semantic from the noisy frames. To overcome such limitations, a fusion based feature extraction can be used in future.

7. Conclusion

Identifying moments in long, untrimmed videos using machine translation queries is a beneficial and demanding endeavour in fine-grained video retrieval. Locating the moments/instances in a video sequence using image queries is investigated and a model is proposed for temporal localization. Two novel algorithms, IMSA and contrastive learning with GMM were proposed to determine alignment locate the temporal bounds. ActivityNet and Charades-STA Benchmark datasets are given as input to the proposed model and the results are evaluated based on precision and recall under IoU curve. It introduces the IMSA transformer and contrastive learning with GMM algorithms, achieving an impressive 86% accuracy on ActivityNet and Charades-STA Benchmark datasets, surpassing convolutional networks in frame extraction for efficient video retrieval. Future research directions include exploring real-world applications like video surveillance, content recommendation, and summarization, alongside enhancing the model’s robustness in handling diverse frame conditions and improving semantic information detection through fusion-based feature extraction methods.

References

Araujo and Girod

, Large-scale video retrieval using image queries, IEEE Transactions on Circuits and Systems for Video Technology 28(6) (2017), 1406–1420.

Chen

Jiang

Sahli

, Transformer encoder with multi-modal multi-head attention for continuous affect recognition, IEEE Transactions on Multimedia 23 (2020), 4171–4183.

Chen

Kornblith

Norouzi

Hinton

, A simple framework for contrastive learning of visual representations, in: International Conference on Machine Learning, 2020 November, pp. 1597–1607. PMLR. doi: 10.48550/arXiv.2002.05709.

Chen

Y.W.

Tsai

Y.H.

Yang

M.H.

, End-to-end multi-modal video temporal grounding, Advances in Neural Information Processing Systems 34 (2021), 28442–28453.

Chen

Wang

Bai

Castañón

Saligrama

, Probabilistic semantic retrieval for surveillance videos with activity graphs, IEEE Transactions on Multimedia 21(3) (2018), 704–716.

Megala

Prabu

, State-of-the-art in video processing: Compression, optimization, and retrieval, Turkish Journal of Computer and Mathematics Education 12(5) (2021), 1256–1272. ISSN: 1309-4653. doi: 10.17762/turcomat.v12i5.1793.

Gao

Sun

Yang

Nevatia

, Tall: Temporal activity localization via language query, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 5267–5275.

Gao

, Learning video moment retrieval without a single annotated video, IEEE Transactions on Circuits and Systems for Video Technology 32(3) (2021), 1646–1657.

Grill

J.B.

Strub

Altché

Tallec

Richemond

Buchatskaya

Valko

, Bootstrap your own latent-a new approach to self-supervised learning, Advances in Neural Information Processing Systems 33 (2020), 21271–21284.

10.

Guo

M.H.

T.X.

Liu

J.J.

Liu

Z.N.

Jiang

P.T.

, Mu, Zhang

S.H.

Martin

R.R.

Cheng

M.M.

S.M.

, Attention mechanisms in computer vision: A survey, Computational Visual Media, 2022, 1–38.

11.

Fan

Xie

Girshick

, Momentum contrast for unsupervised visual representation learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9729–9738. doi: 10.48550/arXiv.1911.05722.

12.

Jaiswal

Babu

A.R.

Zadeh

M.Z.

Banerjee

Makedon

, A survey on contrastive self-supervised learning, Technologies 9(1) (2020), 2.

13.

Lei

Berg

T.L.

Bansal

, Tvr: A large-scale dataset for video-subtitle moment retrieval, in: European Conference on Computer Vision, Springer, Cham, 2020 August, pp. 447–463.

14.

Kim

Kwon

Wang

Kwak

Cho

, Relational self-attention: What’s missing in attention for video understanding, Advances in Neural Information Processing Systems 34 (2021), 8046–8059.

15.

Kordopatis-Zilos

Papadopoulos

Patras

Kompatsiaris

, Visil: Fine-grained spatio-temporal video similarity learning, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6351–6360.

16.

Zhou

Yang

, Sea: Sentence encoder assembly for video retrieval by textual queries, IEEE Transactions on Multimedia 23 (2020), 4351–4362.

17.

Lin

Duan

L.Y.

Wang

Bai

Lou

Chandrasekhar

Huang

Kot

Gao

, HNIP: Compact deep invariant representations for video matching, localization, and retrieva, IEEE Trans. Multimed 19 (2017), 1968–1983.

18.

Mallick

A.K.

Mukhopadhyay

, Video retrieval using salient foreground region of motion vector based extracted keyframes and spatial pyramid matching, Multimedia Tools and Applications 79(37) (2020), 27995–28022.

19.

Megala

Swarnalatha

, Efficient high-end video data privacy preservation with integrity verification in cloud storage, Computers and Electrical Engineering 102 (2022), 108226.

20.

Peng

Ngo

C.W.

, Clip-based similarity measure for query-dependent clip retrieval and video summarization, IEEE Transactions on Circuits and Systems for Video Technology 16(5) (2006), 612–627. doi: 10.1109/TCSVT.2006.873157.

21.

Rahmani

Zargari

, Temporal feature vector for video analysis and retrieval in high efficiency video coding compressed domain, Electronics Letters 54(5) (2018), 294–295.

22.

Shen

Hong

Zhang

Tian

Wang

, Video retrieval with similarity-preserving deep temporal hashing, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 15(4) (2019), 1–16. doi: 10.1145/3356316.

23.

Srihari

R.K.

Zhang

Rao

, Intelligent indexing and semantic retrieval of multimodal documents, Information Retrieval 2(2) (2000), 245–275.

24.

Wang

Cheng

Liu

, DCT-net: A deep co-interactive transformer network for video temporal grounding, Image and Vision Computing 110 (2021), 104183.

25.

Yang

Zhang

, Local correspondence network for weakly supervised temporal sentence grounding, IEEE Transactions on Image Processing 30 (2021), 3252–3262.

26.

You

Sun

Iyyer

, Hard-coded gaussian attention for neural machine translation, arXiv preprint arXiv:2005.00742, 2020.

27.

Song

Wang

Huang

, Intra-and inter-modal multilinear pooling with multitask learning for video grounding, Neural Processing Letters 52(3) (2020), 1863–1879.

28.

Zbontar

Jing

Misra

LeCun

Deny

, Barlow twins: Self-supervised learning via redundancy reduction, in: International Conference on Machine Learning, 2021 July, pp. 12310–12320. PMLR. doi: 10.48550/arXiv.2103.03230.

29.

Zhang

Sun

Jing

Nan

Zhen

Zhou

J.T.

Goh

R.S.M.

, Video corpus moment retrieval with contrastive learning, in: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021 July, pp. 685–695.

30.

Zhang

Sun

Jing

Zhou

J.T.

, The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions, arXiv preprint arXiv:2201.08071, 2022.

31.

Zhou

Chen

Xiao

, Deep learning research with an expectation-maximization model for person re-identification, IEEE Access 8 (2020), 157762–157772.

32.

Zeng

Liu

Zhang

, Improving Video Moment Localization with Stacked Collaborative Transformer Network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

33.

Chen

Wang

, Contrastive Learning for Video Moment Localization, in: Proceedings of the International Conference on Computer Vision (ICCV), 2023.

Stacked collaborative transformer network with contrastive learning for video moment localization

Abstract

Keywords

1. Introduction

2.1 Sentence query-based video retrieval

2.2 Action recognition

3. Proposed methodology

4. Contrastive learning for Spatio-temporal Localization

4.1 Moment localization with contrastive learning

4.4 GMM parameters estimation

5.1 Dataset

Table 4 Dataset description. Dataset Activity Net captions Charades-STA SumMe Number of videos 14950 8354 25 Resolution 480 × 270 1280 × 720 Average Video length 42 s 30 s 6 min Frames per Second (FPS) 30 FPS 30FPS 30 FPS Activity Outdoor activity Indoor activities Outdoor activity

Table 7 Performance measures. No. of queries FLOPs Parameters Top-1 Top-5 1 248.6 G 27.8 M 50.2 78.5 2 89.7 G 23.7 M 51.4 79.4 4 51.4 G 22.3 M 51.7 79.9 6 45.6 G 21.7 M 53.4 89.2 8 33.8 G 20.7 M 55.1 80.6 10 31.6 G 20.3 M 52.3 80.2

References

Table 7
Performance measures.

No. of queries FLOPs Parameters Top-1 Top-5

1 248.6 G 27.8 M 50.2 78.5

2 89.7 G 23.7 M 51.4 79.4

4 51.4 G 22.3 M 51.7 79.9

6 45.6 G 21.7 M 53.4 89.2

8 33.8 G 20.7 M 55.1 80.6

10 31.6 G 20.3 M 52.3 80.2