Research on prevention and control tracking system based on 3D face recognition

Abstract

In order to improve the tracking effect of prevention and control, this paper combines three-dimensional face recognition technology and posture recognition method to build a prevention and control tracking system, and designs a single-stage human target detection and posture estimation network based on thermal infrared images. By using infrared recognition methods, the face and human body shape can be directly recognized, overcoming the problem of traditional visual recognition methods being easily occluded. The network can simultaneously complete two tasks of human target detection and pose estimation in a multi-task manner. Moreover, this paper uses the knowledge distillation strategy to train a lightweight model to further reduce the amount of model parameters to improve the inference speed. In addition, this paper uses a single-stage human target detection and posture estimation network to judge the effect of prevention and control tracking. Through the prevention and control tracking effect test, it can be seen that the prevention and control tracking system based on 3D face recognition proposed in this paper can effectively improve the prevention and control tracking effect.

Keywords

Three-dimensional face recognition prevention and control tracking

1. Introduction

Safe City is a key business system in the construction of e-government in recent years. The urban alarm and monitoring system is the core of the construction of a safe city and an important part of the social security prevention and control system. Moreover, the construction of a safe city has become an important project to provide security and related services for citizens [1]. The main body of a safe city is to use modern information and communication technology to achieve unified command, rapid response, coordinated operations, and improve the efficiency of public security work, so as to meet the needs of dynamic security management and timely and effective crime fighting under modern economic and social conditions, strengthen China’s urban security capabilities, and speed up the construction of urban security systems [2]. It is difficult to adapt to the traditional mode of combating, preventing and managing. In order to adapt to the development needs of the new situation, we must take the road of strengthening the police with science and technology and preventing and controlling with science and technology, among which the construction of urban video surveillance system is an important means [3].

The biggest difference between GIS and other systems is the ability to store and analyze spatial data. The video data of video surveillance systems are mostly geospatial, and the surveillance systems that manage and maintain these video data have complex spatial topological relationships. In addition, shortest path search is a very important function in GIS spatial analysis [4]. The shortest path search algorithm is also being used more and more widely. In addition to the traditional services such as GPS navigation services and urban electronic map path finding services, urban pipe network planning, communication systems, and safe city supervision systems also introduce shortest path search methods. technology [5]. The shortest path not only refers to the shortest distance in the general geographical sense, but can also be extended to other metrics, such as time, cost, and line capacity. Correspondingly, the shortest path problem becomes the fastest path problem, the lowest cost problem, and so on. In the process of driving, it is also necessary to calculate the driving route in front of the vehicle in real time. Therefore, the shortest path analysis should not only be able to characterize the actual access network, but also have a fast enough response speed [6]. Specifically for the GIS-based public security video surveillance system, when a crime occurs, the system can accurately determine the alarm point and the location of the crime scene, automatically notify the relevant police units around the crime scene, and quickly generate the shortest driving route from the criminal’s escape direction. One of the key issues in the field of high-efficiency can improve the police’s rapid response capability and overall command and combat capability in handling cases. Therefore, it has become a necessary and inevitable trend to introduce the core function of GIS-spatial analysis into the video surveillance system [7].

The end result of the underlying data curation is a road network dataset with an advanced connectivity model capable of showing complex details such as multi-model transportation networks, and a rich model of network properties. This model can help model network resistance, network constraints, and network layers. Such network datasets are constructed from simple features such as lines and points and turns. Network datasets provide a powerful management structure for creating, editing and maintaining network data [8]. In the research, a network dataset will be established at all levels of roads and camera locations in Kaifeng, to maintain the connectivity of network elements, and to build an appropriate network model, which will lay a solid foundation for the subsequent optimal path analysis and the realization of the real-time tracking monitoring module. basis [9].

Anti-terrorism and anti-riot robot is a kind of high-tech equipment that integrates mechanical, electronic, sensor, computer, intelligent control, communication and signal processing technology, robotics and other scientific knowledge [10]. It is a high-tech intelligent platform. It can operate reliably for a long time in various complex environments, such as radiation sites, high temperature environments, rugged terrain, etc., and has incomparable advantages in tasks such as reconnaissance, surveillance, and danger elimination. After expanding the automatic targeting system of weapons, it can directly attack terrorists and complete the anti-terrorist task more effectively. At the same time, it can stay in a hidden place for a long time. When terrorists appear, automatic long-range sniping of terrorists can be realized. In addition, its extended robotic arm grasping system can exclude dangerous items when entering the working environment, preventing the possibility of the danger caused by the relevant personnel directly eliminating it [11]. The research and design of new anti-terrorism and anti-riot robots can not only help relevant personnel eliminate hidden safety hazards, combat the arrogance of terrorism, and create a safe social environment, but also improve national competitiveness and comprehensive strength in science and technology. Science and technology are the primary productive forces. In the future, the competition between countries will mainly be based on scientific and technological strength, and the future war will be unmanned war. Robots will be widely used in the military field [12], then national defense technology will become the decisive factor for the outcome of wars. Therefore, it is also necessary to develop intelligent anti-terrorism and anti-riot robots to lay a solid scientific and technological foundation for future military competition [13].

The development of video surveillance system is getting faster and faster. In the beginning, most of them were single-view monitoring. For the areas that need to be monitored, special operators are responsible for managing and viewing the surveillance video. With the rapid development of social life in recent years, the requirements for monitoring systems are increasing day by day. A large number of cameras are used in all aspects of social life. Security, bank monitoring, campus community security, transportation, and smart city construction all require a powerful monitoring system to complete. A large number of cameras bring a large amount of data, and how to apply these data to contribute to human social life is also an urgent problem to be solved [14]. Due to the improvement of the processing speed of the computer itself and the change of the nature of social work, people’s requirements for computer intelligence are also higher. It is the development trend of the current monitoring system to intelligently complete many complicated monitoring tasks in these massive data. Traditional video surveillance systems are mostly centralized surveillance processing platforms [15]. When multiple cameras work at the same time, only the cameras distributed in different locations and different perspectives are responsible for monitoring their own areas, and the video processing work often requires a central system to complete, which increases the network bandwidth requirements, and at the same time, because there is no data exchange and fusion between cameras, a lot of usable information is lost virtually. This results in low utilization and waste of information resources [16].

When carrying out facial recognition work, the application of dynamic facial recognition in security monitoring systems is mainly reflected in the fact that the faces in the picture are always in a non cooperative state, and the monitoring camera is required to respond in a timely manner. Based on the current situation of dynamic facial recognition monitoring systems, the most common device is the high-definition facial recognition monitoring system. Its main application process is to compare the field of view of the monitoring device with the relevant facial data in the black-and-white list database of the deployed personnel, and determine whether the personnel within the field of view are deployed or key personnel. During the operation of this device, automatic analysis can be carried out by actively collecting video images. By comparing faces in real-time and dynamically, the alarm system will be automatically activated when a specific target is detected. At this time, relevant personnel in the monitoring room or command center can pay attention to the alarm personnel through relevant control measures. Based on the current application of dynamic facial recognition monitoring systems, the data capacity of the blacklist/control personnel list database involved can reach hundreds of thousands. The currently common facial recognition monitoring systems are mainly applied in public places such as subways, airports, and train stations. By deploying dynamic facial recognition monitoring systems at the main entrances and exits of these key places, it can effectively meet the security and prevention needs of public security departments, civil aviation, railways and other organizational units. At the same time, this dynamic facial recognition system can also be widely used in household safety and other aspects, providing protection for people’s lives and property.

Video surveillance is a common manifestation of security systems. With continuous development, security systems have now achieved digital and intelligent development. The equipment used is constantly being updated and the functions are becoming more and more perfect, which has been welcomed by the monitoring industry. However, in terms of the application effect of security systems, it is not possible to confirm the physical objects in the monitoring scene. In the entire monitoring scene, characters are the main object and are the core of judging the monitoring scene. By monitoring the characters, alarms are triggered, and external chain sensors are used to judge the physical objects in the scene. However, it cannot effectively identify the changes in the activities of the characters in the video.

To improve the real-time monitoring effect of public safety, a single-stage human object detection and pose estimation network based on thermal infrared images has been designed, which can simultaneously complete both human object detection and pose estimation tasks through multitasking. Due to the limited onboard computing resources of the tracking system, a knowledge distillation strategy is adopted to train a lightweight model, further reducing the number of model parameters to improve inference speed. Therefore, it is possible to achieve human target detection and tracking in both time and space, and effectively improve the system’s computational speed.

A real-time human target detection and pose estimation network based on thermal infrared images was designed for the prevention and control tracking system. The network can simultaneously obtain the two-dimensional coordinates of human targets and key points in the image through infrared images. On this basis, a knowledge distillation strategy is adopted to train a lightweight model, further improving the running speed on the model platform. To obtain the three-dimensional coordinates of human key points, a method for estimating the three-dimensional coordinates of human key points based on circular region sampling was designed. The three-dimensional coordinates of human key points were estimated using two-dimensional coordinates and depth images, and simulation experiments were conducted under laboratory conditions to verify the feasibility of the research content.

This paper combines three-dimensional face recognition technology and gesture recognition method to build a prevention and control tracking system to improve the prevention and control tracking effect.

2. Posture recognition for prevention and control tracking personnel

2.1 Representation and coding method of human body in thermal infrared image

Figure 1.

The method of this article can be summarized as shown in Fig. 1.

Aiming at the characteristics of thermal infrared images, a method of using key points to uniformly represent human targets and two-dimensional human pose was designed. The target detection task and two-dimensional human pose estimation task were unified into a key point detection task. For this representation method, a key point encoding method based on thermal maps and associated embedded maps was designed.

Transforming two different types of tasks into the same or similar expressions for encoding can help improve the reuse rate of network models, thereby reducing the number of model parameters and improving running speed. Meanwhile, using a unified expression for different tasks helps to promote each other and improve the accuracy of both tasks. When encoding borders, transform the object detection problem into a keypoint estimation problem. Unify the representation of human body targets and key points. Specifically, for datasets with K-class human keypoints, the problem will be transformed into a keypoint estimation problem with K $+$ 3 class keypoints.

The image to be detected may contain multiple human bodies, and each human body instance can be composed of a bounding box and a set of human pose information.

Human targets are usually represented by a rectangular frame, as shown in Fig. 1. The four sides of the rectangular border completely enclose all pixels in the image that belong to the same body. For a human target, it can be expressed as $B=\{x,y,w,h\}$ . Among them, $x$ and $y$ represent the pixel coordinates of the center point of the human target, respectively, and $w$ and $h$ represent the width and height of the frame, respectively.

The head of the human body is represented by 2 key points, the parts represented by these key points and their connection are shown in Fig. 2a.

Figure 2.

Representation method of human target and human pose.

For a dataset with K classes of human key points, the problem is transformed into a key point estimation problem with K $+$ 3 classes of key points. Figure 2b shows this representation, where the three green points are the three key points representing the human target, and the red points are the key points representing the human posture.

When constructing the training set, the whole training set contains the annotation information of images and human targets and postures. For the training set $D=\{{E^{i},J^{i},B^{i}}\}_{i=1}^{N}$ , $N$ represents the number of samples in the training set, and $J^{i}$ and $B^{i}$ represent the set of human key points and the set of bounding boxes of the ith sample, respectively. The set $G^{i}$ of all key points in the i-th image $E^{i}$ can be expressed as:

$\displaystyle G^{i}=J^{i}+X^{i}=\{{g_{1}^{i},\ldots,g_{K+3}^{i}}\}$ (1)

Among them, $X^{i}$ represents the set of border key points transformed from the set of borders $B^{i}$ , and $g^{i}$ represents the set of i-th key point labels in the dataset.

The function of the key point heat map is to roughly locate each key point, which is usually generated by a two-dimensional Gaussian kernel function. For the key point $g=(u,v)$ , the process of generating the key point heat map can be described as:

$\displaystyle hp(x,y;g)=\exp\left({-\frac{(x-u)^{2}+(y-v)^{2}}{2\sigma^{2}}}\right)$ (2)

Among them, ( $x, y$ ) represents the pixel coordinates in the heat map, and $\sigma$ represents the width parameter of the Gaussian kernel function. When there are multiple key points of the same type in an image, the pixel value of hp needs to take the maximum value.

The function of the accuracy compensation map is to compensate the coarse positioning result to achieve the effect of precise positioning. Because the model downsamples the image during inference, the size of the model’s predictions often does not match the size of the original image. After the model downsamples the image, the transformation process of the key point g can be expressed as:

$\displaystyle g^{\prime}=({u^{\prime},v^{\prime}})=\left({\frac{u}{\alpha},% \frac{v}{\beta}}\right)$ (3)

Among them, $g^{\prime}$ represents the pixel coordinates of the key point $g$ in the image after downsampling, and $\alpha$ and $\beta$ represent the downsampling ratio in the width and height directions of the image, respectively. The inference result of the key point is calculated from the heat map of the key point output by the model. The inference coordinate is a quantized coordinate, that is, the inference coordinate is an integer coordinate, and the quantization method is usually rounded down. The process can be expressed as:

$\displaystyle g^{\prime\prime}=\lfloor{g^{\prime}}\rfloor=\left({\left\lfloor{% \frac{u}{\alpha}}\right\rfloor,\left\lfloor{\frac{v}{\beta}}\right\rfloor}\right)$ (4)

The inference result $g^{\prime\prime}$ is restored to the coordinates of the original image size according to the formula to obtain $g^{\prime\prime\prime}$ :

$\displaystyle g^{\prime\prime\prime}=\left({\alpha\left\lfloor{\frac{u}{\alpha% }}\right\rfloor,\beta\left\lfloor{\frac{v}{\beta}}\right\rfloor}\right)$ (5)

Since there may be a non-divisible relationship between the original coordinates ( $u, v$ ) of the key point and the downsampling ratio, there may be a case of $g^{\prime\prime\prime}\neq g$ . The function of the precision compensation map is to eliminate the error between $g^{\prime\prime\prime}$ and $g$ .

The grid is the low-resolution pixel coordinate system after downsampling. The red circle is the key point is the coordinate $g^{\prime}$ after the downsampling of the coordinate g, and there is an error $(\Delta x,\Delta y)$ between it and the coordinate $g^{\prime\prime}$ after the quantization operation. In order to predict the result more accurately, this part of the error needs to be compensated during inference. The coordinate of the compensation value rg in the precision compensation diagram is $g^{\prime\prime}$ , and the calculation process can be expressed as:

$\displaystyle rg=(\Delta x,\Delta y)=\left({\frac{u}{\alpha}-\left\lfloor{% \frac{u}{\alpha}}\right\rfloor,\frac{v}{\beta}-\left\lfloor{\frac{v}{\beta}}% \right\rfloor}\right)$ (6)

2.2 Human target detection and posture estimation network based on thermal infrared images

The working method of the network in this paper is shown in Fig. 3. The network is mainly divided into two important parts: the backbone network and the head network. The backbone network is constructed with an improved hourglass network, and the head network is designed based on multi-task learning. According to the characteristics of this network, this section designs a loss function for multi-task learning. In the model training process, the knowledge distillation strategy is used to train the lightweight model to improve the running speed of the model, which is used for deployment in the UAV platform with limited computing resources.

Figure 3.

Schematic diagram of the working method of the network proposed in this paper.

Each hourglass module is divided into multiple stages to process feature maps of different scales, and the N-order hourglass module is denoted by $hg_{N}$ . Figure 4a shows the first-order hourglass module, which is divided into upper and lower branches. If it is assumed that the input feature map is $F_{in}\in R^{H\times W\times C_{in}}$ , the size of the desired output feature map is $H\times W\times C_{\textit{out}}$ , and the upper branch extracts features at the original scale, which is mainly composed of residual modules in series, and the size of the output feature map is $H\times W\times C_{\textit{out}}$ . The lower branch first downsamples once through the max pooling layer to obtain a feature map of size $(H/2)\times(W/2)\times C_{in}$ . Feature extraction and dimension transformation are then performed through multiple residual modules and convolutional layers of $1\times 1$ . After that, the feature map is upsampled by the nearest neighbor value to restore the size of the feature map, and the feature map output by the lower branch is obtained. The size of the output is also consistent with the expected output feature map size, and finally the part in the green dashed box of the lower branch of the $hg_{1}$ module is replaced by a complete $hg_{1}$ module (Fig. 4b). According to this rule, a high-order hourglass module can be formed by continuously nesting the low-order hourglass modules, and the N-order hourglass module is shown in Fig. 4c.

Figure 4.

Hourglass module structure of different orders.

The designed head network module is a single-output multi-output structure, which is mainly divided into two branches: the human body posture branch and the target frame branch. The human body posture branch is responsible for the human key point estimation task, and the target detection branch is responsible for the detection task of the human target frame.

Figure 5.

Loss function residual network structure.

The residual module is an important component of the hourglass module, and its effectiveness directly affects the performance of the entire network. Figure 5a shows the SandGlass module, which not only uses depthwise separable convolution to reduce computational complexity, but also adjusts the order of convolutions in the inverted residual module to improve module performance. After inputting the feature map into the SandGlass module, features are first extracted through a 3 $\times$ 3 convolution layer and ReLU6 is used as the activation function. Then, the first 11 $\times$ 1 convolution is used to reduce the dimensionality of the features and output them linearly. Then, the second 1 $\times$ 1 convolution is used to expand the dimensionality of the features and ReLU6 is used to activate them. Finally, a 3 $\times$ 3 convolution is used to integrate the features, and a short connection structure is added to the input features to obtain the final output of the module. Compared with the inverted residual module, the SandGlass module performs identity mapping and spatial transformation in higher dimensions, which can effectively alleviate the problems of information loss and gradient confusion. For human object detection and key point detection tasks, in order to obtain more accurate coordinate values, the model needs to be able to balance global human information and local details. The main idea of the GC module (Globalcontextblock) is to explore the weight relationship between different information in the feature map through attention mechanism. Figure 5b shows the network structure of the GC module, which combines the advantages of NLNet and SENet mechanisms. It has both the global context modeling ability of NLNet and the lightweight ability like SENet. Therefore, the GC module can be used to further enhance the context fusion ability of the SandGlass module. Figure 5c shows the structure after integrating the SandGlass module and the GC module. After the input features undergo a second 3 $\times$ 3 deep convolution in the SandGlass module, a GC module is used for feature extraction, and then short connections are added to the input module’s data to obtain the final output. This article uses the improved residual module mentioned above to replace the ordinary residual module in the hourglass module to reduce the computational complexity of the network. Finally, this chapter constructs the backbone network using the lightweight improved hourglass module as the basic unit.

The structure of the human target detection branch is shown in Fig. 5.

Figure 6.

Human object detection of head network module.

As shown in Fig. 6, he branch has three branches to deal with the three border key points of the upper left corner, the lower right corner and the center point respectively. Among them, the structure of the output module is the same as that of the human body posture estimation branch. The branches responsible for the upper left corner and the lower right corner of the border have a Top-left corner pooling module and a Bottom-right corner pooling module respectively. The left figure of Fig. 6 shows the structure of the upper left pooling module. For the feature map F of the output module, first, the feature maps $F_{t}$ and $F_{l}$ are generated by two convolution modules as the two inputs of Toppooling and Leftpooling. Then, after the upper left corner pooling layer, two feature maps T and L are generated. The calculation methods for the pixels $t_{ij}$ and $l_{ij}$ in T and L are as follows:

$\displaystyle t_{ij}=\left\{{{\begin{array}[]{ll}{\max({f_{t_{ij}},t_{(i+1)j}}% )}&{\text{if }i<H}\\ {f_{t_{fj}}}&\text{otherwise}\\ \end{array}}}\right.$ (7) $\displaystyle l_{ij}=\left\{{{\begin{array}[]{ll}{\max({f_{l_{ij}},t_{i(j+1)}}% )}&{\text{if }j<W}\\ {f_{l_{iW}}}&\text{otherwise}\\ \end{array}}}\right.$ (8)

Among them, $f_{t_{ij}}$ and $f_{l_{ij}}$ represent the pixel values of $F_{t}$ and $F_{l}$ whose coordinates are ( $i, j$ ), respectively. The module finally adds the two feature maps of T and L to obtain the final output $O_{TL}$ of the upper left corner pooling module. Compared with the common methods such as max pooling and average pooling, the corner pooling method makes the model pay more attention to the features of the target corners. For object detection tasks, it is easier to obtain accurate bounding boxes from the corner features of objects.

The designed model can detect the human target in the image and the two-dimensional posture of the human body at the same time, which belongs to the multi-task learning method. Therefore, a multi-task loss function is required to guide the model optimization during model training. The multi-task loss function designed in this section is:

$\displaystyle L=L_{\textit{pose}}+\alpha L_{\textit{bbox}}+\beta L_{\textit{% off}}+\gamma L_{\textit{tag}}$ (9)

The loss function is mainly composed of 4 parts, where $L_{\textit{pose}}$ is the loss function of the heat map of the key points of the human body, and $L_{\textit{bbox}}$ is the loss function of the heat map of the key points representing the target frame, $L_{\textit{off}}$ is the key point coordinate compensation loss function, $L_{\textit{tag}}$ is the grouping loss function, and $\alpha,\beta,\gamma$ is three hyperparameters, representing the weight of different parts of the loss in the entire loss. The specific calculation methods of each loss function are introduced as follows:

(1) Loss function of heat map of human key points

$\displaystyle L_{\textit{pose}}=\frac{1}{K}\sum\limits_{k=1}^{K}\sum\limits_{i% =1}^{H}\sum\limits_{j=1}^{W}\|{m_{\textit{kij}}-\hat{m}_{kj}}\|^{2}$ (10)

$L_{\textit{pose}}$ adopts the MSE loss function, Eq. (10) is the specific calculation method of $L_{\textit{pose}}$ , where $K$ is the number of human key point categories, $m_{kj}$ and $\hat{m}_{kj}$ respectively represent the predicted value and the true value of the k-th key point heat map at coordinates ( $i, j$ ). The role of this loss function is to guide the output of the network to fit the human key point heatmap generated by the labels.

(2) Loss function of frame key point heatmap

$\displaystyle L_{\textit{bbax}}=\frac{-1}{N}\sum\limits_{c=1}^{3}\sum\limits_{% i=1}^{H}\sum\limits_{j=1}^{W}\left\{{{\begin{array}[]{ll}{({1-y_{\textit{cij}}% })^{\alpha}\log({y_{\textit{cij}}})}&\text{if }\hat{y}_{\textit{cij}}=1\\ {({1-\hat{y}_{\textit{cij}}})^{\beta}({y_{\textit{cij}}})^{\alpha}\log({1-y_{% \textit{cij}}})}&\text{otherwise}\\ \end{array}}}\right.$ (11)

Among them, $y_{\textit{cij}}$ is the predicted value of the coordinate ( $i, j$ ) in the heat map of the c-th frame key point predicted by the model, $\hat{y}_{\textit{cij}}$ is the true value of the heatmap generated by the bounding box label at coordinates ( $i, j$ ), and ${N}$ is the number of objects in the image. In addition, there are two hyperparameters $\alpha$ and $\beta$ in the formula, among which $\alpha$ is used to control the loss weight of difficult and easy samples, so that when the predicted value $y_{\textit{cij}}$ deviates from the true value, a larger loss value can be obtained. For the case where the true values of the heatmap around the center and corner of the border are in the interval (0, 1), $\beta$ can better adjust the loss weight at these positions.

(3) Loss function of coordinate compensation

$\displaystyle L_{\textit{off}}=\frac{1}{N}\sum\limits_{n=1}^{N}\sum\limits_{c=% 1}^{K+3}{\textit{SmoothL1}}({o_{c}^{n},\hat{o}_{c}^{n}})$ (12) $\displaystyle\textit{SmoothL1}(x)=\left\{{\begin{array}[]{ll}0.5x^{2}&\text{if% }|x|<1\\ |x|-0.5&\text{otherwise}\\ \end{array}}\right.$ (13)

Equation (12) is the calculation method of the coordinate compensation loss $n$ , where $L_{\textit{off}}$ is the number of objects in the image, and K $+$ 3 represents the number of all key point categories including the key points of the human body and the key points of the target frame. $o_{c}^{n}$ and $\hat{o}_{c}^{n}$ denote the predicted and ground truth values of the c-th keypoint compensation for the n-th target. The loss adopts the SmoothL1 loss function, and Eq. (13) is the calculation method of the SmoothL1 loss function.

(4) Loss function of grouping

$\displaystyle{L}_{\textit{tag}}=\lambda_{1}L_{\textit{pull}}+\lambda_{2}L_{% \textit{push}}$ (14) $\displaystyle\bar{h}_{n}=\frac{1}{K_{n}+3}\sum\limits_{k=1}^{K_{n}+3}h_{nk}$ (15) $\displaystyle L_{\textit{pull}}=\frac{1}{N}\sum\limits_{n=1}^{N}\sum\limits_{k% =1}^{K_{n}+3}(\bar{h}_{n}-h_{{nk}})^{2}$ (16) $\displaystyle L_{\textit{push}}=\frac{1}{N^{2}}\sum\limits_{n}^{N}\sum\limits_% {n^{\prime}}^{N}\exp\left\{-\frac{1}{2\sigma^{2}}(\bar{h}_{n}-\bar{h}_{n^{% \prime}})^{2}\right\}$ (17)

The backbone network of the teacher model set up in this paper consists of 6 hourglass modules. Since each hourglass module has the same format, each hourglass module can be connected to a complete header network module for output. The output of the head network module in the middle part is different from the final output of the model, and the output of the middle stage only plays a supervisory role in the model training stage. During the deployment and inference stages of the model, the head network modules in the intermediate stages can be cropped, and only the last head network module is retained as the final output. As the data is transmitted to the model backend, the location of key points will become more and more accurate. Therefore, when training the teacher model, a head network module is deployed for each hourglass module, and the network is optimized by a coarse-to-fine multi-branch supervision method, which helps to improve the effect of each stage of the network. Based on this supervision method, the loss function $L_{\textit{all}}$ in the training phase of the teacher model is calculated in the following way:

$\displaystyle L_{\textit{all}}=\sum\limits_{t=1}^{T}\delta_{t}L_{t}$ (18)

Among them, $T$ represents the number of head network modules set in the network, $L_{t}$ is the loss function output by the t-th hourglass module, and the hyperparameter $\delta_{t}$ is the weight set for the output of different stages.

The task of the knowledge distillation stage is to train the student model. The schematic diagram of this stage is shown in Fig. 7. The backbone network of the student model is constructed with two hourglass modules. The loss function used in the knowledge distillation stage is different from the loss function used in the separate training model. It is calculated by the following formula:

$\displaystyle L_{\textit{distill}}=\lambda L_{\textit{hard}}+(1-\lambda)L_{% \textit{soff}}$ (19)

The loss function is divided into two parts, one part is the loss value $L_{\textit{hard}}$ calculated using the hard label and the output of the student model, and the other part is the loss value $L_{\textit{soft}}$ calculated using the soft label and the output of the student model, both parts are calculated using Eq. (18). $\lambda$ is a hyperparameter, which is used to adjust the supervision strength of the two supervision methods, the soft label and the hard label, so that the model can not only learn the label information of the sample through $L_{\textit{hard}}$ , but also learn the generalization ability of the teacher model through $L_{\textit{soft}}$ .

Figure 7.

Schematic diagram of the knowledge distillation method.

Figure 8.

Schematic diagram of direct fusion of thermal infrared image and depth image.

2.3 3D posture estimation of suspicious persons

As shown in Fig. 8, the resolution of thermal infrared image is 640 $\times$ 480, and the resolution of depth image is 960 $\times$ 540. If the two images are converted into the same resolution by interpolation and then directly aligned, there will be obvious errors in the corresponding positions of the two images, and this error cannot be solved by methods such as position compensation or affine transformation. Therefore, it is necessary to register the two images and calculate their pixel correspondences.

Therefore, if the corresponding pixels are obtained in the infrared image and the depth image, the two images need to be registered. The purpose of registration is to construct a transformation matrix ${\rm{\bf W}}^{\prime}$ such that the following equation holds:

$\displaystyle[{{\begin{array}[]{*{20}c}{u_{ir}}&{v_{ir}}&1\\ \end{array}}}]^{T}={\rm{\bf W}}^{\prime}[{{\begin{array}[]{*{20}c}{u_{\textit{% depth}}}&{v_{\textit{depth}}}&1\\ \end{array}}}]^{T}$ (20)

Among them, $({u_{ir},v_{ir}})$ and $({u_{\textit{depth}},v_{\textit{depth}}})$ are the pixel coordinates in the thermal infrared image and the depth image, respectively. Through calibration, the internal parameter matrix of the infrared camera is ${\rm{\bf K}}_{ir}$ , and the transformation from the image coordinate system of the infrared camera to the camera coordinate system can be expressed as:

$\displaystyle[{{\begin{array}[]{*{20}c}{x_{ir}}&{y_{ir}}&{z_{ir}}&1\\ \end{array}}}]^{T}=z_{ir}{\rm{\bf K}}_{ir}^{-1}[{{\begin{array}[]{*{20}c}{u_{% ir}}&{v_{ir}}&1&{1/z_{ir}}\\ \end{array}}}]^{T}$ (21)

Similarly, the internal parameter matrix of the depth camera is ${\rm{\bf K}}_{\textit{depth}}$ , and the transformation from the image coordinate system of the depth camera to its camera coordinate system can be expressed as:

$\displaystyle[{{\begin{array}[]{*{20}c}{x_{\textit{depth}}}&{y_{\textit{depth}% }}&{z_{\textit{depth}}}&1\\ \end{array}}}]^{T}=z_{\textit{depth}}{\rm{\bf K}}_{\textit{depth}}^{-1}[{{% \begin{array}[]{*{20}c}{u_{\textit{depth}}}&{v_{\textit{depth}}}&1&{1/z_{% \textit{depph}}}\\ \end{array}}}]^{T}$ (22)

The transformation matrix from the depth camera coordinate system to the infrared camera coordinate system is M, then the transformation relationship between the two can be expressed as:

$\displaystyle[{{\begin{array}[]{*{20}c}{x_{ir}}&{y_{ir}}&{z_{ir}}&1\\ \end{array}}}]^{T}={\rm{\bf M}}[{{\begin{array}[]{*{20}c}{x_{\textit{depth}}}&% {y_{\textit{depth}}}&{z_{\textit{depth}}}&1\\ \end{array}}}]^{T}$ (23)

Combining Eqs (20)–(23), we can get the relation:

$\displaystyle z_{ir}[{{\begin{array}[]{*{20}c}{u_{ir}}&{v_{ir}}&1&{1/z_{ir}}\\ \end{array}}}]^{T}=z_{\textit{depph}}{\rm{\bf K}}_{ir}{\rm{\bf MK}}_{\textit{% depth}}^{-1}[{{\begin{array}[]{*{20}c}{u_{\textit{depth}}}&{v_{\textit{depth}}% }&1&{1/z_{\textit{depth}}}\\ \end{array}}}]^{T}$ (24)

Since the infrared camera and the depth camera are fixed together and the distance between them is very close, $z_{ir}=z_{\textit{depth}}$ can be considered approximately, and the above formula can be simplified as:

$\displaystyle[{{\begin{array}[]{*{20}c}{u_{ir}}&{v_{ir}}&1&{1/z_{ir}}\\ \end{array}}}]^{T}={\rm{\bf W}}[{{\begin{array}[]{*{20}c}{u_{\textit{depph}}}&% {v_{\textit{depph}}}&1&{1/z_{\textit{depth}}}\\ \end{array}}}]^{T}$ (25)

Through the coordinate system conversion relationship, the conversion relationship from the depth camera coordinate system to the infrared camera coordinate system can be obtained as follows:

$\displaystyle{\rm{\bf M}}=\left[{{\begin{array}[]{*{20}c}{{\rm{\bf R}}_{% \textit{depph}_{i}r}}&{{\rm{\bf T}}_{\textit{depth}_{i}r}}\\ 0&1\\ \end{array}}}\right]=\left[{{\begin{array}[]{*{20}c}{{\rm{\bf R}}_{ir}{\rm{\bf R% }}_{\textit{depph}}^{-1}}&{{\rm{\bf T}}_{ir}-{\rm{\bf R}}_{ir}{\rm{\bf R}}_{% \textit{depth}}^{-1}{\rm{\bf T}}_{\textit{depth}}}\\ 0&1\\ \end{array}}}\right]$ (26)

Therefore, the coordinate mapping relationship from the depth image to the thermal infrared image can be expressed as:

$\displaystyle[{{\begin{array}[]{*{20}c}{u_{ir}}&{v_{ir}}&1&{1/z_{ir}}\\ \end{array}}}]^{T}$ (27) $\displaystyle{}={\rm{\bf K}}_{ir}\left[{{\begin{array}[]{*{20}c}{{\rm{\bf R}}_% {ir}{\rm{\bf R}}_{\textit{depph}}^{-1}}&{{\rm{\bf T}}_{ir}-{\rm{\bf R}}_{ir}{% \rm{\bf R}}_{\textit{depph}}^{-1}{\rm{\bf T}}_{\textit{depph}}}\\ 0&1\\ \end{array}}}\right]{\rm{\bf K}}_{\textit{dephh}}^{-1}[{{\begin{array}[]{*{20}% c}{u_{\textit{depph}}}&{v_{\textit{depph}}}&1&{1/z_{\textit{deph}}}\\ \end{array}}}]^{T}$

A set of two-dimensional human key points in the image is $H=\{{h^{1},\ldots,h^{K}}\}$ , where $h^{i}=({u^{i},v^{i}})$ represents the coordinates of the i-th key point of this person in the image. The mean $\bar{s}$ of all points in point cloud $S^{\prime}$ is calculated:

$\displaystyle\bar{s}=\frac{1}{M}\sum\limits_{m=1}^{M}{s_{m}}$ (28)

For the point set $S^{i}$ of the i-th key point, calculate the Euclidean distance between all points and the mean $g$ , where the Euclidean distance from the j-th point $s_{j}^{i}$ to $\bar{s}$ is:

$\displaystyle\textit{dist}({s_{j}^{i},\bar{s}})=\|{s_{j}^{i}-\bar{s}}\|_{2}$ (29)

The closest point to point $\bar{s}$ in point set $S^{i}$ is $s_{\textit{nearest}}^{i}$ , and the calculation method of this point is:

$\displaystyle s_{\textit{nearest}}^{i}=\mathop{\arg\min}\limits_{s_{j}^{i}}({% \textit{dist}({s_{j}^{i},\bar{s}})})$ (30)

The first $t$ points of the point closest to $\bar{s}$ in the point set $S^{i}$ form a new three-dimensional point set $S^{it}=\{{s_{1}^{it},\ldots,S_{t}^{it}}\}$ , and these $t$ points need to meet the conditions:

$\displaystyle\textit{dist}({s_{j}^{it},s_{\textit{nearest}}^{i}})<\delta$ (31)

The points that do not meet this condition are replaced by $s_{\textit{nearest}}^{i}$ , and the final estimated result of the three-dimensional coordinates of the i-th key point is:

$\displaystyle p^{i}=\frac{1}{t}\sum\limits_{j=1}^{t}{s_{j}^{it}}$ (32)

By performing the above operations on all the two-dimensional key points $H=\{{h^{1},\ldots,h^{K}}\}$ of a human body, the set $P=\{{p^{1},\ldots,p^{K}}\}$ of K three-dimensional key points can be obtained, that is, the estimation result of the three-dimensional coordinates of all the key points of the human body.

3. Research on prevention and control tracking system based on 3D face recognition

Infrared facial images mainly depict the temperature distribution of the face, with skin temperatures near blood vessels slightly higher than those far away.

The skin temperature of blood vessels. Different faces have different temperature distributions. Infrared thermal imagers have high temperature sensitivity and can clearly express temperature distribution. Infrared recognition will not be affected by facial camouflage. If the distribution of blood vessels on the face remains unchanged, the results of infrared facial images will not change.

In the case of only one visible light camera for visible light facial recognition images, if using photos of the target object.

It is possible to achieve successful recognition. Infrared imaging has been introduced, as different photos can be easily distinguished from real people due to the difference between the photos and the real face of the thermal radiation model.

A single-stage human object detection and pose estimation network based on thermal infrared images, which can simultaneously complete both human object detection and pose estimation tasks through multitasking. Using knowledge distillation strategy to train lightweight models, further reducing the number of model parameters to improve inference speed.

In order to improve the prevention and tracking effect of 3D face recognition, this paper constructs the process of face capture for prevention and control as shown in Fig. 9a. Face recognition technology generally uses a face feature selection algorithm based on statistical learning, learning based on massive portrait data, and using the test set to test the impact of the selected different features on the portrait comparison effect while performing feature selection. The best portrait feature is automatically selected from a large number of portrait features according to the test results. Using the automatic optimal feature selection algorithm can reduce the dimension of the feature vector used to describe the portrait information, greatly improve the speed of portrait comparison, and make the algorithm suitable for massive portrait comparison scenes. The specific implementation process is shown in the Fig. 9b.

Figure 9.

Prevention and control tracking system based on 3D face recognition.

As shown in Fig. 9b, facial images were captured by monitoring hardware arranged at various locations. A human model was constructed using the infrared imaging method and encoding method proposed in this paper. The collected facial images were compared with standard data and blacklist databases, and the collected threshold was compared with the standard threshold. If the standard threshold was exceeded, it may indicate that a task affecting public safety has been discovered. An alarm device was triggered to notify security personnel to take appropriate strategies.

For autonomous crossing of narrow passages in any direction. Due to the constraint of the visual angle of the visual sensor, it is not possible to obtain a full view image of the rotor, so it is not possible to narrow the passage of irregularly shaped buildings. To address this issue, it is possible to equip the rotor system with more angle visual sensors and monitoring devices, enabling the system to obtain a sufficiently large sensory field of view, and further research on detection and control strategies for narrow passages in different orientations, enhancing the detection and crossing capabilities of the rotor system for more complex narrow passages.

The experimental environment for this article is as follows:

This system uses PIH-301 gimbal, with motion parameters of: horizontal rotatable angle of 350^∘, vertical rotatable angle of $\pm$ 60^∘, horizontal rotation speed of 6^∘/s, vertical rotation speed of 3^∘/s. Therefore, when setting path keypoints, the accuracy in the horizontal direction is 6^∘ and the accuracy in the vertical direction is 3^∘.

For video surveillance modules, choosing the appropriate development tool will greatly reduce the burden on developers. Developers commonly use three video capture development tools. They are VFW, SDK library functions provided by the acquisition card, and DirectShow, respectively.

The depth sensor model used is IntelRealSenseL515, which captures depth information through laser scanning and is suitable for use in low light scenarios. This chapter also designed and produced a gimbal device that can control the pitch angle of two cameras and increase their visual perception.

In order to test the effectiveness of the human target detection and pose estimation network designed in this paper, a dataset of thermal infrared human body was independently constructed. The algorithm proposed in this paper was compared with existing two-stage methods in experiments, and the results were analyzed.

The model construction and training are completed under the PyTorch deep learning framework. Before training, the data needs to be partitioned first. In this paper, 80% of the dataset is randomly used as the training set and 20% as the testing set. Using Adam as the optimizer and setting the initial learning rate to 0.0015, the learning rates were reduced to 0.00015 and 0.000015 respectively during the 400th and 520th training rounds.

Table 1

The effect of the prevention and control tracking system based on 3D face recognition

Number	Prevention and control tracking	Number	Prevention and control tracking	Number	Prevention and control tracking
1	83.19	23	87.57	45	84.24
2	85.49	24	81.70	46	81.89
3	88.22	25	82.28	47	88.04
4	84.27	26	83.94	48	80.42
5	84.40	27	84.99	49	80.19
6	85.70	28	87.95	50	86.00
7	80.81	29	88.59	51	85.44
8	88.86	30	85.30	52	86.98
9	88.88	31	82.82	53	83.36
10	84.18	32	81.54	54	82.20
11	89.08	33	82.58	55	88.71
12	86.11	34	87.67	56	87.64
13	87.63	35	84.75	57	88.15
14	82.04	36	81.02	58	84.02
15	85.15	37	87.42	59	86.50
16	83.87	38	81.11	60	89.26
17	87.83	39	89.19	61	87.03
18	82.74	40	88.95	62	84.93
19	81.76	41	89.29	63	81.11
20	83.05	42	87.88	64	80.26
21	89.68	43	84.12	65	83.39
22	86.42	44	83.04	66	88.25

Before conducting the experiment, it is necessary to calibrate the infrared camera and depth camera using the binocular camera calibration tool in Matlab, which can calculate the internal parameters and relative position relationship of the two cameras simultaneously. To verify the effectiveness of three-dimensional coordinate estimation of human key points, due to the lack of publicly available datasets of thermal infrared images and depth images, this paper conducted three-dimensional pose estimation experiments using self collected thermal infrared images and depth image data, and calibrated the three-dimensional coordinates of some human key points based on the two-dimensional key point coordinates of the human body in the depth map.

The effect of the prevention and control tracking system based on three-dimensional face recognition proposed in this paper is verified, the practical effect of the prevention and control system is counted, and the statistics are carried out through multiple groups of face recognition effects. Evaluate using expert evaluation methods and a percentage based scoring system, the face recognition prevention and control evaluation effect shown in Table 1 is obtained.

From Table 1, it can be seen that the expert evaluation of the prevention and control tracking effect on 3D facial recognition is above 80 points, and the recognition effect is relatively obvious.

Using 20% randomly partitioned images from a self built dataset as the test set, and comparing it with existing two-stage (YOLOv3 $+$ SHN-6) human object detection and pose estimation methods. In the two-stage algorithm testing, YOLOv3 is first used to detect human targets in the image, and then the detection results are sent to the SHN model for human pose estimation. The comparison results are shown in Table 2.

Table 2

Comparison of method performance

Network model	Backbone network order	AP.5	AR.5	FPS
YOLOv3 $+$ SHN-6	6	0.902	0.973	2.752
The method of this article	6	0.895	0.969	6.346

Compared with the two-stage method YOLOv3 $+$ SHN-6, which also contains 6 hourglass modules, the model in this article has similar accuracy and recall indicators, but its operating frequency is approximately twice that of the latter. It can be seen that the model in this article meets practical requirements.

Through the prevention and control tracking effect test conducted in Table 1, it can be seen that the prevention and control tracking system based on 3D face recognition proposed in this paper can effectively improve the prevention and control tracking effect.

The system in this article adopts an embedded polyp construction method, which can promote the scalability of the system and apply the method in multiple fields, facilitating the further promotion and use of the method in this article.

4. Conclusion

In order to effectively crack down on illegal crimes, cut off some criminal suspects’ escape routes after the incident, and significantly reduce the number of wanted fugitives across the country, the Ministry of Public Security has attached great importance to the arrest of fugitives by local public security organs at all levels. By using infrared recognition methods, the face and human body shape can be directly recognized, overcoming the problem of traditional visual recognition methods being easily occluded They include information comparison, call monitoring of the fugitives’ communication tools, work on the fugitives’ social relations and family members, launching psychological offensives against the fugitives, and increasing publicity and publicity. Controlling and cracking down on crimes is the main target of the construction of the public security prevention and control system, while illegal behavior is a secondary aspect, which also includes assisting the public security organs in catching criminal suspects at large through the public security prevention and control system. This paper combines three-dimensional face recognition technology and posture recognition method to build a prevention and control tracking system. Through the prevention and control tracking effect test, it can be seen that the prevention and control tracking system based on 3D face recognition proposed in this paper can effectively improve the effect of prevention and control tracking.

The main method of this article is to use infrared recognition for facial and human pose recognition, overcoming the problem of traditional visual schemes being unable to cope with occlusions. The system in this article is greatly affected by temperature during application. If there is interference similar to human body temperature, it will affect the recognition effect. Therefore, subsequent research will combine infrared recognition and visual recognition technology to further improve the adaptability of the method in this article.

References

Deng

. Blended emotion in-the-wild: Multi-label facial expression recognition using crowdsourced annotations and deep locality feature learning. International Journal of Computer Vision.2019; 127(6): 884-906.

Ali

Tian

Din

, et al. Classical and modern face recognition approaches: A complete review. Multimedia Tools and Applications.2021; 80: 4825-4880.

Cowen

Keltner

Schroff

, et al. Sixteen facial expressions occur in similar contexts worldwide. Nature.2021; 589(7841): 251-257.

Gogić

Manhart

Pandžić

, et al. Fast facial expression recognition using local binary features and shallow neural networks. The Visual Computer.2020; 36(1): 97-112.

Carletti

Greco

Percannella

, et al. Age from faces in the deep learning revolution. IEEE Transactions on Pattern Analysis and Machine Intelligence.2019; 42(9): 2113-2132.

Gupta

Thakur

Kumar

. 2D-human face recognition using SIFT and SURF descriptors of face’s feature regions. The Visual Computer.2021; 37(3): 447-456.

Nirmala Sreedharan

Ganesan

Raveendran

, et al. Grey wolf optimisation-based feature selection and classification for facial emotion recognition. IET Biometrics.2018; 7(5): 490-499.

Barat

Wirth

Duhamel

. Face cells in orbitofrontal cortex represent social categories. Proceedings of the National Academy of Sciences.2018; 115(47): E11158-E11167.

Ghorbandaei Pour

Taheri

Alemi

, et al. Human-robot facial expression reciprocal interaction platform: Case studies on children with autism. International Journal of Social Robotics.2018; 10: 179-198.

10.

Johnston

Chazal

. A review of image-based automatic facial landmark identification techniques. EURASIP Journal on Image and Video Processing.2018; 2018(1): 86.

11.

Zhang

Verma

Tjondronegoro

, et al. Facial expression analysis under partial occlusion: A survey. ACM Computing Surveys (CSUR).2018; 51(2): 1-49.

12.

Kang

Jia

Gao

, et al. Extracting human emotions at different places based on facial expressions and spatial clustering analysis. Transactions in GIS.2019; 23(3): 450-480.

13.

Graziani

Melacci

Gori

. Coherence constraints in facial expression recognition. Intelligenza Artificiale.2019; 13(1): 79-92.

14.

Chandra

Bedi

. Survey on SVM and their application in image classification. International Journal of Information Technology.2021; 13(5): 1-11.

15.

Erol

Majumdar

Benavidez

, et al. Toward artificial emotional intelligence for cooperative social human-machine interaction. IEEE Transactions on Computational Social Systems.2019; 7(1): 234-246.

16.

Khan

Hussain

Usman

. Reliable facial expression recognition for multi-scale images using weber local binary image based cosine transform features. Multimedia Tools and Applications.2018; 77: 1133-1165.