Neural network-based motion vector estimation algorithm for dynamic image sequences

Abstract

With the rapid development of deep learning, convolutional neural networks have gradually become the main means to extract features of dynamic image sequences. The motion vector estimation algorithm, as the key to the stability of image sequences, directly affects the performance of image stabilization systems, so the motion estimation algorithm for convolutional neural networks is necessary. The study proposes an improved convolutional neural network based on loss-free function, and applies it to the extraction of dynamic image features. On this basis, the motion estimation algorithm is then optimised by combining grey-scale projection and block matching methods. The experimental results show that the new loss-free function-based convolutional neural network has better recognition capability with an error rate of only 15% in dynamic image recognition. The accuracy of the optimised motion estimation algorithm is as high as 95.1% with a PSNR value of 16.636, which is higher than that of the traditional grey-scale projection algorithm. In terms of video processing, the improved algorithm has a higher PSNR value than the search block matching method, the bit-plane matching method and the full search block matching method, with a higher steady image accuracy and high operational efficiency, providing a new research idea for the improvement of motion estimation algorithms. In general, the proposed algorithm is a significant improvement over the current mainstream algorithms in terms of image accuracy, processing performance and number of operations, and it provides a new research idea for the improvement of motion estimation algorithms.

Keywords

CNN dynamic image sequences grey-scale block matching motion

1. Introduction

In recent years, image feature extraction using deep convolutional neural networks has developed rapidly [1]. However, current convolutional neural networks are increasingly difficult to train. Not only does the number of parameter tuning increase, but the convergence of the loss function is also poor [2]. In addition, with the increasing quality of videos, how to implement the combination of convolutional neural networks and motion vector estimation algorithms has gradually become the focus of research. The estimation accuracy and detection rate of motion vector estimation algorithms, which are key factors directly affecting the stability of image sequences, are limited by the algorithms themselves. Therefore, this study optimizes the problems of existing image feature extraction techniques by performing effective feature extraction of dynamic image sequences and analyzing the optimization methods of motion estimation algorithms, so as to improve the accuracy of image feature extraction and enhance the operational performance of motion estimation algorithms.

The article is divided into five main parts. The second part is Related Work, which introduces and sorts out the main research achievements in the field of the research object in recent years. The third part is Methods, which introduces the design and principle of the algorithm and model. The fourth part is Experiments, which introduces the performance experiment results of the proposed model. The fifth part is the Conclusion, which summarizes the research results.

2. Related work

In recent years, professionals at home and abroad have conducted in-depth research on image sequence motion vector estimation algorithms using neural networks. Li et al. [3] transformed the kernel size estimation problem into a regression problem in order to recover the blurred kernel of single blurred images and a potentially clear images, and solved it by constructing a convolutional neural network, which showed that the method accurately estimated the motion blurred kernel size and was able to obtain better images. Li et al. [4] proposed a new dynamic image representation method based on convolutional neural networks by merging convolutional features across a maximum pool of frames and using a linear support vector machine to train the classifier. Experiment proved its high performance. Saif et al. [5] employed convolutional neural networks as the main architecture for traffic estimation, image segmentation and action recognition, and processed feature maps using normalised vector maps with additional spatial and temporal information in the fusion layer. Experiments showed that the method improved the efficiency of behaviour recognition. Ma and Zhao [6] used convolutional neural network representations as an overall image representation and topologically retrieved similar looking keyframes from the topology map and used the sharpness metric to select keyframes, effectively avoiding blurring of keyframes. Bu [7] addressed the problem of poor human motion gesture recognition by extracting feature maps using deep convolutional neural networks, which had lower dimensionality and higher discrimination than traditional feature images extracted from fully connected layers. Peng et al. [8] proposed a new network structure allowing an arbitrary number of frames as network input and used a feature connectivity layer that combined motion information and appearance, while introducing a module consisting of a temporal pyramid pooling layer and an encoding layer. This structure was proved to have high operational efficiency. Suzuki and Ikehara [9] used implicit learning of inter-frame motion to generate intermediate frames directly and learn the differences among them through residual learning. The results showed that the method improved recognition performance and can avoid uncertainty in motion estimation.

Jesi et al. [10] addressed the overfitting problem of traditional hidden conditional neural field classifiers by extracting feature vectors based on a neural network with weighted deviation means. Experiment showed that it reduced execution time and improved accuracy. Ahmed and Khot [11] proposed a directional edge recovery method with a video encoder compression scheme to improve the spiral pixel reconstruction algorithm, and experimental results showed that the method was able to improve video coding efficiency. Traver and Paredes [12] used convolutional neural networks to regress motion parameters in order to perform iamge motion estimation, and used synthetic image morphing for experiments on existing image datasets. The results showed that the method the accuracy obtained was better than that of Cartesian images. Duan et al. [13] proposed the application of deep convolutional neural networks to single-person pose estimation for the problem of low accuracy of multi-person pose estimation. It was used to generate human heat maps, which improved the generalisation of extracted features. Shao et al. [14] developed a new neural network structure for transient changes, temporal correlation and interference of unknown factors on image sequences. Errors were eliminated through temporal correlation, and experimental results showed that its convergence speed is fast. Wu et al. [15] proposed a feature fusion strategy integrating image recognition and motion estimation to address the problem of poor real-time performance of neural networks. Experiments showed that it could extract more representative features for multi-task learning. Sun et al. [16] used deep convolutional neural networks for feature extraction to cope with noisy factors and unfamiliar scenes, and experimental results showed that the method was robust to interference from noisy factors.

Recently, there are also some researches on Dynamic Image Sequences. Farokhah [17] studied the performance of facial emotion recognition model and proposed a dynamic image sequences recognition method based on machine learning. The experimental results show that the method is practical. Tsintotas et al. [18] and its research partners applied Dynamic Image Sequences technology to robot positioning and mapping, and their new method was tested on seven public datasets, and the results showed that this method was superior to the original method. In summary, convolutional neural networks are capable of feature extraction and classification of dynamic image sequences. Most researchers have proposed new convolutional neural network models to address the problems of poor accuracy and inefficiency, and the accuracy of convolutional neural network in some application scenarios is improved. However, the accuracy still has room for improvement and it is difficult for such techniques to take into account the indexes of operation accuracy, resource saving and time spending, while less research has been done on the motion vector estimation algorithm based on neural networks. Therefore, in this study, the motion vector estimation algorithm of neural networks is improved and applied to image feature extraction in order to improve the performance of the algorithm image feature extraction.

3. Convolutional neural network-based motion vector estimation algorithm for dynamic image sequences

3.1 Dynamic images based on convolutional neural networks

With the continuous improvement of video and image quality used by people, users have higher requirements for Dynamic images than ever. In order to meet the requirements of users, deep convolution neural network is a feasible technology. Deep convolutional neural networks have strong image feature extraction capability and can be applied to various image recognition algorithms. The alternating superposition of pooling and convolutional layers forms the structure of deep convolutional neural networks. The feature map in the convolutional layer is formed by the convolutional kernel of that layer after sliding convolutional filtering of the output image of the previous layer. The convolutional layer is immediately followed by the pooling layer, where the down-sampling operations of the features are performed, including the selection of the maximum pooling and the calculation of the average pooling [19]. The image features from the convolutional layer need to be quantized and then connected to the fully connected layer. Finally the output layer and the features are connected to the classifier and output together into various types of probabilities. The label corresponding to the largest probability is selected among the output probabilities for the final recognition result. The model of a deep convolutional neural network is shown in Fig. 1.

Figure 1.

Convolutional neural network model.

Each layer of the convolutional neural network consists of $n$ blocks of feature images containing individual neurons, and has a strong self-learning capability, fault tolerance and parallel processing capability, enabling direct input of 2D image data, extraction and feature classification of image features through the network. The combination of local perceptual field, weight sharing and downsampling allows the convolutional neural network to maintain a good tolerance to the input data. Local field of perception means that the network layer only connects to neurons in a small domain in the previous layer, rather than to all of them, so that all primary visual features can be accurately extracted, and subsequent layers can then combine and classify them to obtain more advanced features [20]. Weight sharing features means that the current convolutional layer can only be used to extract the same features located at different locations in the previous layer of the network, thus significantly reducing the number of weights and data for training. It also reduces complexity and alleviating the overfitting problem. The downsampling stage enables sub-sampling of the upper layer network feature image blocks, ensuring the valid information and reducing the scale of data processing. In the convolution layer, the feature pixel values are formed by the sliding convolution of the convolution kernel of that layer with the output image of the upper layer, so that the feature pixel values of the feature map are calculated as shown in Eq. (1).

$\displaystyle v\left\{{\begin{array}[]{l}y_{pj}^{l}=f\left({\sum\limits_{i\in M% _{j}^{l-1}}{\sum\limits_{(u,v)\in K^{l}}{W_{ij_{(u,v)}}^{l}\circ X_{pi}^{l-1}(% u+c,r+v)+b_{j}^{l}}}}\right)\\ K=\left({(u,v)\in N^{2}\left|{0\leqslant v<k_{y},0\leqslant u<k_{x}}\right.}% \right)\\ \end{array}}\right.$ (1)

In Eq. (1), $j$ represents the sequence of feature maps, $l$ represents the sequence of convolutional layers, and $b$ represents the bias. $c$ is the current longitudinal feature pixel, $r$ is the horizontal feature pixel, and $W_{ij}^{l}$ is the convolution kernel. $u$ represents the vertical step of the convolution kernel, $v$ is the horizontal step of the convolution kernel. $f$ represents the activation function, and $p$ is the training sample sequence. $N$ represents the number of neurons, and $X$ is a unknown number. The convolution operation occurs when convolving the input feature map. Each pixel value in the pooling layer is formed by pooling window sliding from the convolutional feature map sampled from the output of the previous layer, and its feature pixel value is calculated as shown in Eq. (2).

$\displaystyle\left\{{\begin{array}[]{l}y_{pj}^{l}=\left({\frac{\sum\limits_{(u% ,v)\in S^{l}}{X_{pi}^{l-1}(u+c,v+r)}}{s_{x}\cdot s_{y}}}\right)\\ S^{l}=\left({(i,j)\in N^{2}\left|{0\leqslant v<k_{y}^{l},0\leqslant u<s_{x}^{l% }}\right.}\right)\\ \end{array}}\right.$ (2)

In Eq. (2), $s_{x}^{l}$ and $s_{y}^{l}$ are both downsampling windows. $r$ and $c$ represent the horizontal and vertical feature pixels respectively. $v$ and $u$ represent the vertical and horizontal steps of the pooling window respectively. The average of the pooled area is obtained by adding up all the feature pixel values in the pooling window and dividing them by the total number of pixels. The averaging pooling process is shown in Fig. 2.

Figure 2.

Average pool diagram.

The basic structure of a fully connected layer is a multilayer perceptron, consisting of two layers, an input layer and an output layer, connected by a weight matrix. The input layer of the multi-layer perceptron is formed by vectorising the feature map of the previous layer. The weight matrix is inner-producted with the input vector, then mapped by the activation function, and the result is formed into the output layer. Therefore, the fully connected layer is calculated as shown in Eq. (3).

$\displaystyle y_{pj}^{l}=f\left({\sum\limits_{i=0}^{N^{l-1}}{X_{pj}^{l-1}\cdot b% _{j}^{l}+w_{ji}^{l}}}\right)$ (3)

In Eq. (3), $b$ represents the bias of neurons and $w$ is the weight. Through the feature extraction of the convolutional neural network, the dynamic image is transformed into a feature vector of the fully connected layer, which is finally classified in the classifier, and then the image is classified by the corresponding label with the maximum probability calculated by the classifier.

Convolutional neural networks are trained in a way that at its core uses machine learning methods to extract image features, which are then transformed into convolutional kernels. The study proposes a new loss-free function convolutional neural network for image feature extraction. Each vector is specified to obtain the image training set and vectorisation set, each element of the matrix in the intrinsic plot represents the distance between one of the samples patch and another patch, as shown in Eq. (4).

$\displaystyle\gamma=\sum\limits_{c=1}^{C}{\sum\limits_{i\in S_{c}}{\sum\limits% _{j\in N_{d_{1}}(i)}{\left\|{\overline{Z_{i}}-\overline{Z_{j}}}\right\|}}}^{2}% =2\overline{Z}(D^{w}-S^{w})\overline{Z}^{T}\in R^{k^{2}\cdot k^{2}}$ (4)

In Eq. (4), $\overline{Z_{i}}$ is the vectorised set, $Z$ is the training set and $N_{d_{1}}(i)$ is the set of nearest neighbour ordinal numbers. $D^{w}$ represents the intra-class relationship. $S^{w}$ represents the intra-class connectivity. The resulting intra-class relationship diagonal matrix and intra-class connection relationship matrix are given in Eq (5).

$\displaystyle\left\{{\begin{array}[]{l}S_{ij}^{w}=\left\{{\begin{array}[]{ll}1% ,&\textit{if}\ j\in N_{d_{1}}(i)\ \textit{or}\ i\in N_{{}_{d_{1}}}(j)\\ 0,&\textit{else}\\ \end{array}}\right.\\ D_{ii}^{w}=\sum\limits_{j\neq i}{S_{{}_{i,j}}^{w},i,j=1,2,\ldots,N}\\ \end{array}}\right.$ (5)

In Eq. (5), $S_{ij}^{w}$ is the intra-class connectivity matrix and $D_{ii}^{w}$ is the intra-class relationship diagonal matrix. In the penalty diagram, the elements of the interclass dispersion matrix characterise the distance between the sample patch and its nearest neighbour patch between the other classes and the formula is given in Eq. (6).

$\displaystyle\phi=\lambda=\sum\limits_{c=1}^{C}{\sum\limits_{i\in S_{c}}{\sum% \limits_{j\in P_{d_{2}}(i)}{\left\|{\overline{Z_{i}}-\overline{Z_{j}}}\right\|% }}}^{2}=2\overline{Y}(D^{b}-S^{b})\overline{Y}^{T}\in R^{k^{2}\cdot k^{2}}$ (6)

In Eq. (6), $\phi$ represents the interclass dispersion matrix and $P_{d_{2}}^{(i)}$ is the set of nearest neighbour ordinal numbers. $\overline{Y}$ is the set of relative elements. The equations for the diagonal matrix of dissimilar class relationships and the matrix of dissimilar connection relationships are shown in Eq. (7).

$\displaystyle\left\{{\begin{array}[]{l}S_{ij}^{b}=\left\{{\begin{array}[]{ll}1% ,&\textit{if}\ j\in P_{{}_{d2}}(i)\ i\in P_{2}(j)\\ 0,&\textit{else}\\ \end{array}}\right.\\ D_{ii}^{b}=\sum\limits_{j\neq i}{S_{{}_{i,j}}^{b},i,j=1,2,\ldots,N}\\ \end{array}}\right.$ (7)

In Eq. (7), $S_{ij}^{b}$ denotes the heterogeneous connection relation matrix and $D_{ii}^{b}$ denotes the heterogeneous relation diagonal matrix. Therefore, the formula for solving the set of eigenvectors is given in Eq. (8).

$\displaystyle(\phi-\gamma)v_{i}^{1}=\lambda_{i}v_{i}^{1},i=1,2,3,\ldots,L_{1}$ (8)

In Eq. (8), $\lambda_{i}$ denotes eigenvalue $i$ of the matrix ( $\phi-\gamma$ ), $L_{1}$ denotes the number of convolution kernels in layer $i$ , and $V$ denotes the set of eigenvectors. The feature vectors are reconstructed into a two-dimensional convolution kernel matrix at layer 1, as shown in Eq. (9).

$\displaystyle W_{i}^{1}=\textit{mat}_{k\cdot k}(v_{i}^{1})\in R^{k\cdot k}$ (9)

In Eq. (9), $W_{i}^{1}$ is the MFA convolution kernel $i$ , and the $\textit{mat}(\cdot)$ function represents the two-dimensional matrix. After the first layer is trained, it is then convolved with the original image to obtain the output feature map of the first layer, as shown in Eq. (10).

$\displaystyle I^{2}=I^{1}\circ W^{1}$ (10)

Training the second layer is exactly similar to the first layer, and repeating this process can have the second convolutional layer trained. Then the formula for each group of features is shown in Eq. (11).

$\displaystyle Q_{i}=\left\{{I_{i}^{2}\circ W_{n}^{2}}\right\}_{n=1}^{L},i=1,2,% 3,\ldots L_{1}N$ (11)

In Eq. (11), $Q_{i}$ is the group feature. Therefore, the convolutional neural network model without loss function is shown in Fig. 3.

Figure 3.

Improved convolutional neural network model diagram.

3.2 Motion estimation algorithm based on convolutional neural networks

After extracting the feature of dynamic images by convolutional networks, motion vector estimation is then performed. The main motion estimation algorithms include block matching algorithm, representative point matching method and grey scale projection algorithm, etc. The study focuses on block matching algorithm and grey scale projection algorithm. The block matching algorithm is to divide the reference frame image into a series of non-overlapping macroblocks, which are treated as blocks to be matched, and treat all pixels within the blocks as pixels with the same motion characteristics [21]. The difference between the horizontal and vertical coordinates is the displacement of the matching block, i.e. the translation motion vector of the current frame. The block matching method is shown in Fig. 4.

Figure 4.

Schematic diagram of block matching method.

The absolute mean difference, normalised intercorrelation function and mean square error are the matching criteria of the block matching motion estimation algorithm. They determine the accuracy, data reading complexity, matching operation complexity and memory management complexity of the motion estimation. Based on the matching criterion, the best search path is found. The full search method is the most reliable search method, which will match all pixels within a certain search range and then get the optimal motion vector, which is reliable, simple and accurate, but the computation is too huge. The greyscale projection algorithm mainly reflects the distribution of image greyscale and is applied to two adjacent frames for rough motion vector estimation. Specifically, the current frame and the reference frame are projected, and then the rank correlation between them is calculated separately and the minimum value is found in their correlation curves, and the resulting motion vector is the roughly estimated motion vector. The general steps are image mapping, projection filtering and correlation calculation. Image mapping is the pre-processing of all frames in the image sequence with histogram equalisation, which maps the two-dimensional image grey scale information into two independent one-dimensional information, and then performs the row and column projection of the grey scale values, i.e. accumulates the grey scale values of the pixels in all rows and columns of the image.

$\displaystyle\left\{{\begin{array}[]{l}G_{k}j=\sum\limits_{i}{G_{k}(j,i)}\\ G_{\textit{mean}}=\frac{\sum\limits_{j}{G_{k}\left(j\right)}}{n_{c}}\\ G_{rk}j=G_{k}j-G_{\textit{mean}}\\ \end{array}}\right.$ (12)

In Eq. (12), $G_{k}j$ represents the sum of the grey values of all the pixels in the column $j$ of the frame $k$ , $G_{k}(j,i)$ represents the grey values of the pixels in the column of the $(j,i)$ frame $k$ . $G_{\textit{mean}}$ represents the mean of the column projections, $n_{c}$ represents the number of columns, and $G_{rk}j$ is the normalised projection of the mean value of the column $j$ of the frame $k$ . The resulting projection is then filtered by a cosine square filter to reduce the amplitude of the boundary information, thus preserving the central region waveform and improving the accuracy of displacement vector detection, as shown in Eq. (13).

$\displaystyle G_{k}j=\left\{{\begin{array}[]{ll}G_{rk}j\left\{{1+\cos\left[% \frac{\pi(F-j-1)}{2F}\right]}\right\},&j\in(n_{c}-F,n_{c}),(F,0)\\ 0,&\textit{else}\\ \end{array}}\right.$ (13)

In Eq. (13), $F$ represents the filter width taken and $n_{c}$ represents the number of columns. The column projection curve of the reference frame is correlated with the column projection curve of the current frame. The horizontal motion displacement of the current frame with respect to the reference frame is determined by the unique valley value in the correlation curve, which is calculated as shown in Eq. (14).

$\displaystyle C(w)=\sum\limits_{j=1}^{cl}{\left[{G_{k+1}(w+j-1)-G_{k}(j+m)}% \right]}^{2},(1\leqslant w\leqslant 2m+1)$ (14)

In Eq. (14), $c l$ represents the length of the selected column, $m$ represents the search width of the displacement vector, $G_{k}j$ and $G_{k+1}j$ represent the grey-scale projection values of the column $j$ of the image $k$ of the reference frame and the column $j$ of the image $k+1$ of the current frame, respectively, and $C_{(w)}$ is the calculated value of the column correlation. When $w=w_{\min}$ and $C_{(w)}$ are minimum, the displacement vector of the current frame in the horizontal direction with respect to the reference frame is given in Eq. (15).

$\displaystyle\delta_{x}=(w_{\min}+j-1)-(j+m)=w_{\min}-m-1$ (15)

In Eq. (15), $\delta_{x}$ represents the displacement vector and $\delta_{x}>0$ represents a shift of one pixel to the right with respect to the current value of the reference frame. Therefore, the motion vector of the current frame in the vertical direction with respect to the reference frame is shown in Eq. (16).

$\displaystyle\delta_{y}=(j+m)-(w_{\min}+j-1)=m+1-w_{\min}$ (16)

Figure 5.

Identification error rate change chart of MFA-1 and MFA-2.

In Eq. (16), $\delta_{y}$ represents the motion vector. On the basis of obtaining the displacement vectors in the vertical and horizontal directions, the image sequence is kept stable by moving the image of the current frame by the corresponding pixel distance in the opposite direction of the displacement vector. A rough motion vector is obtained by the conventional grey-scale projection method, and then an adaptive full-search block matching method is applied to obtain an accurate estimate of the direction. The coarse estimated motion vector is first compensated by a backward shift and partitioned with the difference image. After removing the search width around the image, the difference image is divided into 16 macroblocks of 64 $\times$ 64 pixels, and then the sum of the grey values of all macroblock pixel points is obtained. They are then sorted, weighing the estimated motion and accuracy, and 10 macroblocks with a smaller sum of grey values are selected as candidate macroblocks and their starting coordinate positions are recorded. Finally, the feature complexity of the reference frame macroblocks is estimated, and the one with a larger grey scale variation is used as the block to be matched. Considering the accuracy and complexity of the algorithm, the mean squared difference is used to describe the feature complexity of the macroblocks, and the formula for the mean value of the macroblocks is shown in Eq. (17).

$\displaystyle u_{k}=\frac{1}{N_{L}M}\sum\limits_{i=1}^{M}{\sum\limits_{j=1}^{N% }{f_{k}(i,j)}}$ (17)

In Eq. (17), $k$ represents the number of macroblocks, $f_{k}$ is the grey scale value. $M$ and $N_{L}$ are the length and width of the macroblocks, and the macroblock size can be obtained by multiplying the two. The resulting mean squared deviation of the macroblocks is shown in Eq. (18).

$\displaystyle\overline{\sigma_{k}}=\frac{1}{N_{L}M}\sum\limits_{i=1}^{M}{\sum% \limits_{j=1}^{N}{\left({f_{k}(i,j)-u_{k}}\right)}}^{2}$ (18)

In Eq. (18), $\overline{\sigma_{k}}$ represents the macroblock mean squared deviation. Thus, the sum of the rough estimate of the motion vector and the exact estimate of the motion vector gives the global motion vector.

4. Analysis of application effects

Firstly, the experiment and analysis of the dynamic image recognition with the convolutional neural network model based on the loss-free function proposed in the study was carried out. Two MFA convolutional layer networks, namely MFA-1 and MFA-2 were built, containing one and two layers of MFA convolutional layer networks respectively. The number of convolutional kernels in MFA-1 ranged from 4 to 32, while the number of convolutional kernels in the first layer of MFA-2 was the same as the former, and the second layer was set to a fixed number of 8. The parameters $d_{1}$ and $d_{2}$ were 10 and 100 respectively, and $b$ was 2. The above parameters are determined based on the simulation calculation and test before the experiment, and are the settings for optimizing the model performance within the parameter value range. The variation of image recognition rate for different number of MFA convolution kernels is shown in Fig. 5.

As can be seen from Fig. 5, both MFA-1 and MFA-2 have increasing recognition rates as the number of convolutional kernels increases. Among them, MFA-2 reaches the lowest error rate at $L_{1}=$ 16 and $L_{2}=$ 8, and the recognition effect of MFA-2 is significantly better than that of MFA-1. It is because of the difference in the number of convolutional kernels and the deep overlay network which enables the feature map to extract more abstract features layer by layer, which improves the representation of the extracted features and enhances the classification effect. The parameters $d_{1}$ were set from 4 to 15, $d_{2}$ was set from 50 to 6 $C$ , while the intervals of $d_{1}$ and $d_{2}$ were 1 and 50 respectively. The remaining parameters of MFA-2 were kept unchanged, and the effects of the parameters $d_{1}$ and $d_{2}$ on the image classification results are shown in Fig. 6.

Table 1
Recognition accuracy of differen method on ICDAR2017

Comparison method	Recognition accuracy (%)
CNN $+$ Softmax	94.7
HOG $+$ SVM	56.8
Stroke Bank	81.2
MSER	68.5
Feature Pooling Method	79.4
PCANet-2 $+$ SVM	82.9
MFA-2 $+$ SVM	95.1

Figure 6.

Change diagram of MFA-2 recognition error rate.

As can be seen from Fig. 6, the recognition error rate of MFA-2 tends to decrease in general as the parameters $d_{1}$ and $d_{2}$ becoming larger, with the lowest error rate of 15% at $d_{1}$ of 15 and $d_{2}$ of 300. In terms of trend, the recognition error rate did not decrease significantly with larger $d_{1}$ at $d_{1}\geqslant 13$ and $d_{2}\geqslant 250$ . The change in the number of same-class neighbours has a smaller effect on the recognition error rate, while the change in the number of dissimilar neighbours has a larger effect, but in general, the recognition error rate decreased as the number of dissimilar and similar neighbours increased. CNN $+$ Softmax, HOG $+$ SVM, Stroke Bank, MSER, Feature Pooling Method and PCANet-2 $+$ SVM algorithms are compared with MFA-2 convolution with parameters $k=$ 7, $d_{1}=$ 15, $d_{2}=$ 300, $L_{1}=$ 32 and $b=$ 7. For CNN, three convolution layers are set, with convolution numbers of 64, 96. The final fully-connected layer of the CNN is set to 2048 fully-connected neural units, and all convolutional kernels are pre-trained by a self-coding machine and then fine-tuned 300 times using training samples feature Pooling. The method uses pyramidal pooling layers to downsample features, maximizing the spatial pooling units to 7 $\times$ 7, 5 $\times$ 5 and 3 $\times$ 3, with step pools of 2, 4 and 6, and a final fully connected layer of 4096 neural units. ICDAR2017 is chosen for the dataset and the recognition accuracies of the different algorithms are shown in Table 1.

As can be seen from Table 1, the MFA convolutional network algorithm proposed by the Institute achieves an accuracy of 95.1% for extracting image features, which is significantly higher than the recognition accuracy of the other algorithms. In addition the CNN $+$ Softmax algorithm achieves 94.7% recognition accuracy, mainly because it uses three layers of convolution and pooling layers, one more layer than the one proposed by the Institute, so with the increase in the number of convolution layers, the recognition accuracy of the MFA convolutional network will be further improved. The improved motion vector estimation algorithm is then experimentally analysed with the operating system Windows XP, and simulation experiments are carried out via Metalab 7.0, with real dynamic image sequences selected. In order to accurately detect the motion vector of the image, the motion vector of the dynamic image sequence is estimated from coarse to fine, and the resolution of the image is 1920 $\times$ 1080.

Figure 7.

Row column projection curve of reference frame and current frame.

As can be seen in Fig. 7, the minimum value of the column correlation curve is 2 $\times$ 10 ${}^{10}$ , and the minimum value of the row correlation curve is 3 $\times$ 10 ${}^{5}$ , yielding a coarse estimate of the motion vector of (2, 6). The block matching fine estimation is then performed and the local motion vectors calculated by the full search block matching method are shown in Table 2.

Table 2

Motion estimation results of dynamic image sequences

Search starting point	Rough estimation of motion vector	Precise estimation of motion vector	Final motion vector
(18,242)	(0, $-$ 4)	(2,6)	(2,1)
(82,242)	(0, $-$ 6)	(2,6)	(2, $-$ 2)
(82,184)	(0, $-$ 9)	(2,6)	(2, $-$ 4)
(152,242)	(0, $-$ 6)	(2,6)	(2, $-$ 2)
(208,184)	(0, $-$ 6)	(2,6)	(2, $-$ 2)

From Table 2, it can be seen that after median filtering of the five local motion vectors, the mean of the middle three motion vectors after sorting is selected to yield a finely estimated motion vector of (0, $-$ 6), and the global motion vector of (2, $-$ 2) after coarse and fine estimation. The PSNR results of the motion vector estimation after taking the commonly used steady image algorithm are shown in Table 3.

Table 3

Comparison of several motion estimation algorithms

Motion estimation algorithm	PSNR (dB)	Type	Times	Digit
Reference frame and current frame	13.857	N.A	N.A	N.A
Gray projection method	14.352	Addition and subtraction	435562	8
Full search block matching method	16.848	Addition and subtraction	3326528	8
Improved algorithm	16.636	Addition and subtraction	428080	8

Figure 8.

PSNR curve.

As can be seen from Table 3, the PSNR value of the improved algorithm proposed in the study, which performs coarse estimation before fine estimation, is 16.636, which is significantly higher than that of the traditional grey-scale projection algorithm and slightly higher than that of the full-search block matching at 16.537, indicating a higher steady image accuracy. As can be seen from the number of operations, the number of operations of the improved algorithm is 428080, while that of the full-search block matching method is 3326528 and 435562 for the traditional grey-scale projection method, the improved algorithm has a much lower number of operations than the full-search block matching and a lower complexity. The performance of the full search block matching method, the improved algorithm, the search block matching method and the bit-plane matching method were compared, and the PSNR graphs of the four algorithms for video processing are shown in Fig. 8.

As can be seen from Fig. 8, the improved algorithm achieves the highest PSNR value of 30, which is higher than the PSNR values of the other three algorithms, indicating that it is more effective and has higher performance than the other algorithms for processing dynamic image sequences.

5. Conclusion

By combining the improved convolutional neural network with the motion estimation algorithm to improve the stability of dynamic image sequences. The experimental results show that the improved MAF convolutional network model has an increasing recognition rate and a gradually decreasing error rate as the number of convolutional kernels increases, and the recognition error rate decreases as the number of dissimilar and similar neighbours increases. With one less layer of both convolutional and pooling layers, the accuracy of extracting image features is 95.1%, which is slightly higher than the 94.7% of the CNN+Softmax algorithm. The PSNR value of the improved motion vector estimation algorithm is 16.636, which is significantly higher than that of the traditional grey-scale projection algorithm, and slightly higher than that of the full-search block matching algorithm at 16.537, indicating a higher steadystate image accuracy. In terms of the number of operations, the number of operations of the improved algorithm is only 428080, which is much lower than that of the full-search block matching algorithm at 3326528, indicating a lower complexity. In terms of processing dynamic images, the improved motion estimation algorithm has a higher PSNR value than all three commonly used algorithms, indicating a superior processing performance. In summary, this algorithm has higher image accuracy and processing performance, as well as lower number of operations and complexity compared to the main algorithms in the current image feature extraction field, which is an significant improvement to the field. However, the study takes less account of changes in depth of field during the dynamic image transformation process, and further analysis of this is needed.

References

Babaee

Dinh

Rigoll

. A deep convolutional neural network for video sequence background subtraction. Pattern Recogn. 2018; 76(1): 635-649.

Liu

Deng

Chanussot

Hong

Zhao

. StfNet: A two-stream convolutional neural network for spatiotemporal image fusion. IEEE T Geosci Remote. 2019; 57(8): 1-13.

Sang

Yan

Gao

. Motion-blur kernel size estimation via learning a convolutional neural network. Pattern Recogn Lett. 2017; 119(Mar.): 86-93.

Chen

. Multiple feature fusion in convolutional neural networks for action recognition. Wuhan Univ J Nat Sci. 2017; 22(1): 78-83.

Saif

Akib

Khan

Julian

Karmoker

. Aggressive action estimation: A comprehensive review on neural network based human segmentation and action recognition. Int J Educ Manag Eng. 2019; 9(1): 9-19.

Zhao

. Robust topological navigation via convolutional neural network feature and sharpness measure. IEEE Access. 2017; 5(1): 20707-20715.

. Human motion gesture recognition algorithm in video based on convolutional neural features of training images. IEEE Access. 2020; 8(1): 160025-160039.

Peng

Cao

Shen

Liu

Shen

. Temporal pyramid pooling based convolutional neural networks for action recognition. IEEE T Multimedia. 2017; 27(11): 2613-2622.

Suzuki

Ikehara

. Residual learning of video frame interpolation using convolutional LSTM. IEEE Access. 2020; 8(1): 134185-134193.

10.

Jesi

Aslam

Mazharuddin

. Deep learning based dynamic hand gesture recognition with leap motion controller. Int J Adv Trends Comput Sci Eng. 2020; 9(4): 7309-7315.

11.

Ahmed

Khot

. An efficient generalized error concealment in video codec. International J Comput Vision Image Process. 2020; 10(3): 1-28.

12.

Traver

Paredes

. Study of convolutional neural networks for global parametric motion estimation on log-polar imagery. IEEE Access. 2020; 8(1): 149122-149132.

13.

Duan

Wang

Cui

Sang

Sun

. Multi-person pose estimation based on a deep convolutional neural network. J Vis Commun Image R. 2019; 62(Jul.): 245-252.

14.

Shao

Han

Wei

Wang

. A novel recurrent convolutional neural network-based estimation method for switching guidance law. IEEE Access. 2020; 8(1): 10159-10168.

15.

Zhang

Tian

. Simultaneous face detection and pose estimation using convolutional neural network cascade. IEEE Access. 2018; 6(1): 49563-49575.

16.

Sun

Zhu

. A novel convolutional neural network based localization system for monocular images. Int J Softw Sci Comp. 2019; 11(1): 38-50.

17.

Farokhah

. Implementasi convolutional neural network untuk klasifikasi variasi intensitas emosi pada dynamic image sequence. Jurnal RESTI. 2020; 4(5): 1070-1076.

18.

Tsintotas

Bampis

Gasteratos

. Tracking-DOSeqSLAM: A dynamic sequence – Based visual place recognition paradigm. IET Comput Vis. 2021; 15(3): 25-273.

19.

Marban

Srinivasan

Samek

Andeza

Casalsa

. A recurrent convolutional neural network approach for sensorless force estimation in robotic surgery. Biomed Signal Proces. 2019; 50(Apr.): 134-150.

20.

Ranjbar

Singleton

Jackson

Rickertsen

Whitmire

Clark-Swanson

, et al. A deep convolutional neural network for annotation of magnetic resonance imaging sequence type. J Digi Imaging. 2020; 33(1): 439-446.

21.

Zhuang