IMF-MF: Interactive moment localization with adaptive multimodal fusion and self-attention

3.1 Query-awareness for improved retrieval

Description of every video clip using visual and textual modalities as shown in Fig. 1. We combine the modalities using a linear weighted sum rather than joining the two modalities together to create a lengthy given input variable as the network input [31]. The user query informs the adaptive derivation of the weights. In particular, a query highlights verbal conversation content that is anticipated to be absent from video data. In this case, the text element obtained from the text-based method should receive higher priority in fusion. However, some queries should be given more visual weight because they are more appealing visually than textually. Two learnable weights, W _v and W _t which represent the visual and text methods features of a clip, respectively, combine to form the query-dependent fusion function. OPEN IN VIEWER

In order to improve the ability of visual and textual methods to discriminate, firstly combine the input features S _i and V _i into a single dimension. To be more precise, both modalities are first linearly projected to have the same dimensionality. Following the Multi-Modal Transformer’s feature pre-processing [11, 23] each projected feature is added to two self-learning vectors that the information particular to each method and the temporal variation. The initialization of the two vectors referred to as temporal and method embedding, respectively, is random. L _v = 100 as a result of a video’s variable number of clips. A maximum of 100 snippets for every video are used to train the temporal encoding algorithm. Each embedding encodes a time-dependent feature. Zero vectors are used to represent videos with fewer than 100 clips when the embedding for both the temporal and methods is combined. In learning the connection between the clips, the sets of characteristics are then simultaneously fed into a multi-model transformer to produce contextualized visual and textual characteristics. V b = { V m 1 , V m 2 , … , V mn } R n B s = { T m 1 , T m 2 … , T mn } R n(2)

At this point, the clip-level characteristics use video-level data to provide global categorization in addition to encoding intra-modality signals. There should be noted that only temporal encoding is learned because the query is conveyed in text content. If a query can only have 30 tokens, then L _q = 30 embedding are learned in total. I = ( W v × V m + W t × T m ) R Lo(3)

The reason for maintaining distinct token features is that user queries often include data from multiple video clips, and the connection between the clips and tokens must be modeled. To achieve this, a transformer contextualizes each token feature with all the other tokens in the query, similar to how it processes video clips. NetVLAD (network vector of locally aggregated descriptors) is then used to aggregate the transformed query token features and create query-dependent weights using a learned codebook of 33 clusters. This results in a vector that contains the residuals between the user query and codebook, reflecting the entire query globally. To learn the query-dependent weights, a 1-layer FC network with softmax is added on top of NetVLAD (network vector of locally aggregated descriptors). The fused and contextualized-modal feature successfully combines two different modalities based on the query nature. The bi-directional memory-free attention mechanism from [37] is used to localize and add query-aware characteristics to the query-dependent fusion feature of each video clip. This attention mechanism weights clip features by query tokens, resulting in an R^Lo similarity matrix that indicates which clips and tokens are comparable. Ai and j are components of the R^Lo (similarity matrix). L _q , which is the focus of attention, indicates that the i^th clip and the j^th token are comparable. The formula illustrates this similarity- a i , j = w 1 · 1 i + w 2 · j + w 3 ( l i · j )(4) where w₁, w₂, w₃, and R are learnable vectors and denote element-wise accumulation. Video-to-query attention determines how relevant a query token is to a video clip. The attention weight placed on the j^th token by the i^th clip is denoted as A _i , j. Each visited query vector I of the clip is a linear sum of the weights of query-token characteristics. A i , j = e ai j ∑ j = 1 a i j(5)

By query to video attention, the clips that most closely resemble one of the query tokens are given preference. Thus, clips that is completely irrelevant to any token play a reasonably small role in the later stage of feature aggregation. Let B _i stand for the token with the highest overall similarity rating for the i^th clip. The weighting of the clip features in relation to the search perspective yields the participated video vector q _v . B i = max ai , j A i = e b ∑ j = 1 b i j , Q v = ∑ j = 1 Y i · N i(6)

Thus every query-dependent clip feature can have query-aware features added to it, for instance by concatenating the Y _i and N _i from video to query attention and the Q _v from query to video attention. However, since Q _v is a vector that applies to all clips, we adhere to the advice in [37] and append the triplets as query-aware feature learning features for each clip.

A segment or temporal location head is formed by combining a series of clips (C _i … C _e ). The query-aware feature learning feature of each clip is used to predict the start and finish score distributions of a segment. We use a transformer to pre-process query-aware learning features before identifying the inter-clip relationship. To lessen the information loss brought on by history and context, the transformed characteristics are convolved with the characteristics from the preceding stage. Before feeding the feature to ConvSE We empirically sample videos up to a search depth, to increase the sample size and place more emphasis on concrete negative examples.

The proposed model leverages the search perspective for segment localization and ranking by using a two-stage approach. In the first stage, the model uses a bi-directional attention mechanism to learn the relationships between the query and the video segments. This allows the model to identify the video segments that are most relevant to the query, even if they are not directly mentioned in the query. In the second stage, the model uses a ranking algorithm to rank the candidate video segments. The ranking algorithm takes into account the scores from the bi-directional attention mechanism, as well as other factors such as the temporal structure of the video segments. The bi-directional attention mechanism allows the model to learn the relationships between the query and the video segments in both directions. This means that the model can learn the relationships between the query words and the video segments, as well as the relationships between the video segments themselves. We continue to use the to calculate b _y , b _S , and b _Q for query-dependent fusion, using Multi-model transformer and transformer. The fusion weight v, t is derived using NetVLAD. We also learn to combine two vectors, one for each of the two modalities, visual and text [20]. To compute the video similarities the vectors are used directly. The fusion weights v, t are used to further weight the sum of the two similarities scores. It should be noted that this condensed model does not use query-aware feature learning or segment localization heads. Up to a search depth of d = pi + xi, where p is the rank of the video and xi is the extension range, we sample videos from a small training batch during training. When the negative video number is set to 3, it displays the proposed model’s result sum curves for various similarity scoring functions. As we can see, the setting chosen for this work, x = 500, produces the best results. The performances of negative videos that were randomly sampled are consistently worse than the performance of x = 500. It implies how crucial it is to watch some negative videos alongside popular ones.

3.3 Video segment extraction

The proposed model involves parsing the top-k videos suggested by the first-stage search engine and generating B _i and E _o score variances for each of them. The external product of normalized factors is taken, and their affectedness within each video is computed, to produce the mathematical function of segment aspirants. After post-processing, lengthy and brief segment candidates are discarded using non-maximum suppression. The remaining candidates are ranked based on their likelihood, which can be calculated in three different ways. The approach multiplies the segment probability with the score of video similitude inherited from the first-stage search engine, while the second approach ignores the first-stage score and uses the score of video similitude produced by the suggested model and video scoring head. The calculation of video scores is a multi-faceted process that considers several factors. We evaluate query relevance through metrics like textual similarity, temporal overlap, and segment equality. Visual clarity and audio quality, is also weighed. Normalization and aggregation techniques ensure fair comparisons, leading to the identification of relevant segment candidates. Video scores directly influence the ranking of candidates, ultimately enhancing the precision of segment retrieval in our approach. The third approach simply ranks segment candidates according to their probabilities. The proposed model outperforms existing models by using NetVLAD to enrich query representation with contextually pertinent word clusters [33] allowing for better multi-modal similarity scoring. Additionally, it employs ConvSE with discriminative inputs instead of similarity scores. The shared normalization used in this model enables the sampling of hard negative videos from the top-k videos for model training, making it superior to existing models that only use actual videos for training.

4.1 Dataset

TVR [19] –The videos in TVR (TV show retrieval), a closed-world dataset [31, 37] are taken from six long-running TV programs. Videos are on average 79.3 seconds long and come with time-stamped subtitles. Each video has 5 queries created specifically for it, totaling 5,500 queries in the testing set. The segment lasts an average of 9.1 seconds and can last anywhere between 0.29 and 123.02 seconds.

4.2 Implementation

The video is divided and text descriptions are assigned to clips whose timestamps overlap, but there may be multiple videos associated with a single description. The segment length is limited to a range of [L _min , L _max ], with L _min  and L _max  empirically set for each dataset (D _s ). The learning rate is set to 1e-4. There are two cross-entropy loss functions, corresponding to the ML and VS heads of IMF-MF. The weights for moment and video loss are set to 1e-2 and 5e-2, respectively. We only use the moment loss for general and disjoint similarity scoring functions (see Equation (3)), whereas we use the combination of the two for exclusive similarity scoring functions. We employ an early stopping approach and compute the sum of R@1 values at IoU = {0.5, 0.7} after each training epoch. The training only involves queries whose ground-truth videos can be retrieved within the search depth of 100 by the 1st-stage search engine. This is because some queries are ambiguous and could have multiple moments as ground-truths, although only one of them is marked as positive. Involving these queries can negatively impact learning effectiveness. About 0.5% of queries are excluded from training. The training is ended when the value of the sum consecutively drops for three epochs in the validation set. The training time depends on the number of negative videos being sampled. When keeping the number to three, IMF-MF converges in about 7.5 to 9 hours depending on the similarity scoring function for the TVR dataset. The number of unfavorable videos sampled affects training time and convergence, with the model typically converging after 7.5–9 hours on a single NVIDIA quardo RTX 5000 GPU et.

4.3 Evaluation

In this section, we present both qualitative and quantitative outcomes for the TVR, DiDeMo and Charades –STA dataset. Initially, we outline our assessment criteria and subsequently compare our results with those of baseline methods.

The outcomes of various methods on the TVR dataset are shown in Table 1. On both the validation and test sets, the proposed model significantly outperforms all other approaches at all recall levels. The x-axis in the given column graph represents the different evaluation metrics R1, R5, and R10. The y-axis represents the corresponding average recall values for each of these metrics, as obtained through different combinations of VCMR and SVMR models and different IoU thresholds (0.7). The IOU (intersection over union) value is a commonly used evaluation metric in computer vision tasks, such as object detection and segmentation. In the case of the VCMR (video corpus moment retrieval) and SVMR (single video moment retrieval) tasks, IOU is used as a threshold for positive segment candidates. This means that a predicted segment is considered positive if its IOU with the ground truth segment is equal to or greater than the specified threshold. The reason for choosing IOU values of 0.7 or 0.5 may vary depending on the task and the dataset being used. Generally, a higher IOU threshold means that the predicted segment must be more precise and accurate to be considered positive, while a lower IOU threshold is more lenient and allows for more variation in the predicted segments. The choice of IOU threshold depends on the specific requirements of the task and the dataset (D _s ) being used.

The IOU (Intersection over union) value is a commonly used evaluation metric in computer vision tasks, such as object detection and segmentation. In the case of the VCMR (video corpus moment retrieval) and SVMR (single video moment retrieval) tasks, IOU is used as a threshold for positive segment candidates. This means that a predicted segment is considered positive if its IOU with the ground truth segment is equal to or greater than the specified threshold. The reason for choosing IOU values of 0.7 or 0.5 may vary depending on the task and the dataset being used. Generally, a higher IOU threshold means that the predicted segment must be more precise and accurate to be considered positive, while a lower IOU threshold is more lenient and allows for more variation in the predicted segments. The choice of IOU threshold depends on the specific requirements of the task and the dataset being used. Tables 2 and 3 present the comparison of the proposed framework’s performance on the DiDeMo and charades-STA dataset with previous models. In Fig. 2(a), shown moment localization and (b), (c) shown temporal distribution on dataset. It is worth noting that only 50% of the videos have textual descriptions, and most searches rely on visual cues. Hence, the proposed framework only focuses on the visual aspect. Despite not using the text method, the proposed model’s results are better than existing model.

OPEN IN VIEWER

Fig. 2

(a) Prediction visualization for the localization of moments (blue color line) has shown through the use of brackets and lines and also shows video score. (b), (c) Temporal Distribution of moments shown in the video dataset.

The proposed IMF-MF framework outperforms existing state-of-the-art methods on all three benchmark datasets. On the TVR dataset in Table 1, IMF-MF achieves a recall of 9.23% at top-1, compared to 8.56% for the second-best method (HMAN). On the DiDeMo dataset, IMF-MF achieves a recall of 5.37% at IoU = 0.5, compared to 4.12% for the second-best method (HMAN) as shown in Table 2. On the Charades-STA dataset, IMF-MF achieves a recall of 6.18% at IoU = 0.5, compared to 5.71% for the second-best method (HMAN) as shown in Table 3.These results demonstrate that IMF-MF is a promising new method for video moment retrieval. The framework outperforms existing state-of-the-art methods on all three benchmark datasets, and its performance is consistent across different recall levels. Figure 2(a) provides a visual comparison of IMF-MF against baselines (Self-Attention and Bi-directional Attention) for moment localization. The blue line representing IMF-MF exhibits superior localization accuracy, likely due to its effective integration of query context and multi-modal features. Figure 2(b) and (c) illustrate the temporal distribution of moments across the entire video corpus and individual videos, correspondingly, suggesting the model’s generalizability and applicability to diverse content. Overall, these figures demonstrate the efficacy of the proposed IMF-MF approach for accurate and robust video moment localization. The framework’s ability to effectively capture query-specific information, model relationships between query and video features, and fuse visual, audio, and textual information makes it a powerful tool for localizing relevant video segments.