Reliability evaluation of dynamic face recognition systems based on improved Fuzzy Dynamic Bayesian Network

Abstract

The reliability of face recognition system has the characteristics of fuzziness, randomness, and continuity. In order to measure it in unconstrained scenes, we find out and quantify key broad-sense and narrow-sense influencing factors of reliability on the basis of analyzing operation states for six dynamic face recognition systems in the practical use of six public security bureaus. In this article, we propose a novel evaluation method with True Positive Identification Rate in dynamic and M:N mode and create a novel evaluation model of system reliability with the improved Fuzzy Dynamic Bayesian Network. Subsequently, we infer to solve the fuzzy reliability state probabilities of the six systems with Netica and get two most important factors with the improved fuzzy C-means algorithm. We verify the model by comparing the evaluation results with actual achievements of these systems. Finally, we find several vulnerabilities in the system with the least reliability and put forward a few optimization strategies. The proposed method combines advantages of the improved fuzzy C-means model with those of the dynamic Bayesian network to evaluate the reliability of the dynamic face recognition systems, making the evaluation results more reasonable and realistic. It starts a new research of face recognition systems in unconstrained scenes and contributes to the research on face recognition performance evaluation and system reliability analysis. Besides, the proposed method is of practical significance in improving the reliability of the systems in use.

Keywords

Dynamic face recognition system reliability evaluation Fuzzy Dynamic Bayesian Network

Introduction

Nowadays, the public security situation is confronted with two major difficulties. Traditional and non-traditional security threats are intertwined. The complexity of social contradictions and the threat of terrorism are increasing. The situation has brought unprecedented pressure and challenge to public security agency in their work of combating, preventing, managing, and controlling threats. However, the increase in police force is far from enough to meet the actual demand. So the face recognition technology, which can transform science and technology into combat powers directly, has become an important means to solve this problem. Face recognition is a kind of biometric recognition technology based on face feature information. Compared with other biometric technologies, the face recognition technology has unique advantages, such as non-mandatory, contactless, intuitive, convenient, and quick. Accordingly, it has broad application prospects. As one of the most popular biometric technologies, face recognition is a milestone for the advancement of security technology. The dynamic face recognition system is a kind of technical equipment that is especially for public security prevention and control in public areas, which is funded, used, and managed by public security agency. It is a product of the face recognition technology, and it is also a Hand of Midas for the efficiency and value of the public security video surveillance. Its basic principle is to identify target faces automatically in the public security video surveillance. At present, the dynamic face recognition system has become indispensable for public security agency,¹ and it plays an irreplaceable role in public security business.

With the increasing application of the dynamic face recognition system, its reliability has been paid more and more attention by public security agency and scholars. Face recognition is one of the most challenging problems in pattern recognition and computer vision. The reliability of the face recognition system is influenced by many factors, such as power supply, weather, and system performance. Although the face recognition system in constrained scenes has perfect reliability and the recognition accuracy is comparable to human eyes, in unconstrained scenes, for example, the dynamic face recognition system often has unsatisfactory, even greatly discounted reliability. The reliability of the dynamic face recognition system is an important influencing factor of the system effectiveness, and the reliability evaluation of the dynamic face recognition system has become a crucial research topic naturally.

The academic research works on reliability start from World War II. The reliability analysis gradually extends from electronic and aerospace systems to many other fields, such as machinery and communication. The common methods of system reliability analysis include fault tree analysis,² Markov process,³ Monte Carlo simulation,⁴ failure mode effects and criticality analysis,⁵ reliability block diagram,⁶ dynamic fault tree,⁷ fuzzy numbers,⁸ GO methodology,⁹ Petri net,¹⁰ universal generating function,¹¹ binary decision diagrams,¹² Bayesian network,^13,14 and dynamic Bayesian network.^15–18 Most of the methods suppose that the variables of system reliability only have two states (either 1 or 0). However, there may be multiple discrete, even continuous state variables in the reliability of many complex systems. At the same time, because of the complexity and uncertainty of systems, there is fuzziness in variable states of reliability. The fuzzy Bayesian network emerges subsequently. The deficiencies of the fuzzy set theory itself may cause information loss in the process of fuzzy Bayesian network inferring and solving. In addition, the above-mentioned research works do not consider broad-sense but narrow-sense influencing factors of reliability. For example, the recognition performance is an important factor that influences the reliability of the dynamic face recognition system.

At present, there are a few relevant research works on the security system. Lv¹⁹ evaluates the reliability of a set of security equipment by testing their lives and fitting their failure functions. The method is quite time-consuming, and it only analyzes internal failure factors. In Qu et al.,²⁰ a failure risk evaluation index system of a video surveillance system and the fuzzy analytic hierarchy process model are created to evaluate the system failure probability, that is, the reliability evaluation of the video surveillance system. The index system is mostly qualitative, and the evaluation method itself is rather subjective.

Aimed at the problems in previous research works, we propose a relatively objective and effective reliability evaluation model with the improved Fuzzy Dynamic Bayesian Network. It is a novel reliability evaluation model with a combination of improved fuzzy C-means and dynamic Bayesian network. We evaluate the reliability of dynamic face recognition systems with the improved Fuzzy Dynamic Bayesian Network model and find out important influencing factors and vulnerabilities, and put forward a few optimization strategies.

The innovations and contributions of this work are summarized as follows:

We find out influencing factors of the reliability of face recognition systems in unconstrained scenes, including both broad-sense and narrow-sense factors, and quantify them scientifically and reasonably.

We propose a novel evaluation method with True Positive Identification Rate (TPIR) of face recognition systems in dynamic and $M : N$ mode.

We create a novel evaluation model of system reliability with the improved Fuzzy Dynamic Bayesian Network, which can dynamically evaluate the reliability of the systems with characteristics of fuzziness, randomness, and continuity. The Membership functions of the improved Fuzzy Dynamic Bayesian Network model are constructed not according to subjective experience but objective sample data. So the proposed model is obviously more advantageous than previous methods.

We conduct evaluation experiments on six dynamic face recognition systems in use and verify the model by comparing the evaluation results with actual achievements of these systems. Experimental and verification results show that our approach is reasonable and realistic.

Approach

The Bayesian network, also known as the belief network, is one of the most effective theoretical models in the field of random uncertainty knowledge representation and inference. It has obvious advantages in logic descriptions of random unascertained failures, while the simple Bayesian network has some limitations, mainly because it can only describe discrete random variables with finite states.

Jousselme et al.²¹ categorize unascertained knowledge into two kinds, which are ambiguity and vagueness. They correspond to random variables and fuzzy variables, respectively. The influencing factors of the dynamic face recognition system reliability are random, fuzzy, and continuous. “Incidence” is used to indicate the possibility of a random event, and the Membership Degree is used to indicate the extent to which a fuzzy event belongs to a certain state. Both the possibility and the Membership Degree are continuous. For example, “high incidence of camera failure” is a mixed event with the characteristics of randomness and fuzziness. The occurrence of this event involves probability and degree. “Incidence” is random and “high” is fuzzy. To describe “high incidence of camera failure,” we need to consider both the probability of “incidence” and the degree of “high.”

In many complicated cases, the variables are all random, fuzzy, and continuous. If we describe this kind of variables with the simple Bayesian network, it is equivalent to assigning all of the degrees to 1 or 0. It means that the node variables either belong to a certain state or do not belong to the state at all, without considering intermediate states of node variables. The fuzzy Bayesian network and the Fuzzy Dynamic Bayesian Network can solve the kind of problems. The former is a model that combines the fuzzy set theory and the simple Bayesian network,^22–29 which is premised on the basis of the hypothesis that the system reliability state is independent of time. The latter is a model that combines the fuzzy set theory and the dynamic Bayesian network,^30–32 which is premised on the basis of the hypothesis that the system reliability state is related to time and changes with time. If we regard the state change process of the system reliability as a series of snapshots, each snapshot describes the state of the system reliability at a specific time. Actually, every snapshot, also known as a time slice, is a fuzzy Bayesian network. The Fuzzy Dynamic Bayesian Network is a special form of the fuzzy Bayesian network, and it consists of a series of fuzzy Bayesian networks. Because the system reliability state is often time-dependent in reality, the Fuzzy Dynamic Bayesian Network is more suitable for system reliability description. But for both the fuzzy Bayesian network and the Fuzzy Dynamic Bayesian Network, there is a major defect in the construction of Membership functions. It may cause inaccuracy in evaluation results.

The improved Fuzzy Dynamic Bayesian Network model consists of the improved fuzzy C-means and the dynamic Bayesian network. Its main idea is to use improved Membership functions in fuzzy sets to represent the states of continuous variables and compute the prior probabilities on the basis of considering degrees. The fuzzy C-means is a mathematical model that is used for the soft classification of samples according to their attributes.^33,34 Because the Membership functions of the improved fuzzy C-means model are constructed according to objective sample data, they have obvious advantages over fuzzy Membership functions that are constructed according to subjective experience in unascertained information processing. The improved fuzzy C-means model is good at processing fuzzy unascertained information, while the dynamic Bayesian network is good at processing random unascertained information. We combine them to create a new approach to evaluating the reliability of complex systems, which is the improved Fuzzy Dynamic Bayesian Network. In this section, we only present the approach of the improved fuzzy Bayesian network, and we extend it on time series to get the improved Fuzzy Dynamic Bayesian Network model in the process of modeling in the next section.

Suppose $u_{1}, u_{2}, \dots, u_{s}, \dots, u_{Q}$ represent $Q$ dynamic face recognition systems that need to be evaluated. Let $U = {u_{1}, u_{2}, \dots, u_{s}, \dots, u_{Q}}$ . $X_{1}, X_{2}, \dots, X_{d}$ are influencing factors of system reliability, namely, evaluation indicators. We regard them as $d$ attribute nodes (root nodes) of the improved fuzzy Bayesian network. $X_{si}$ represents the $i th$ attribute of $u_{s}$ , which is a mixed variable that is both fuzzy and random. Suppose observation times of the $i th$ attribute of system $u_{s}$ are $n_{si}$ and the $j th$ observed value of the $i th$ attribute of system $u_{s}$ is $x_{sij}$ $(s = 1, 2, \dots, Q; i = 1, 2, \dots, d; j = 1, 2, \dots, n_{si})$ .

We need to quantify to what degree $x_{sij}$ belongs to a certain state $Γ_{k}$ before evaluating system $u_{s}$ . Let $Γ = {Γ_{1}, Γ_{2}, \dots, Γ_{k}, \dots, Γ_{C}}$ , which is the set of states, and $Γ_{1}, Γ_{2}, \dots, Γ_{k}, \dots, Γ_{C}$ are ordered and limited, where $C$ is the number of states or classifications. $Γ_{k}$ also represents a mixed event classification. ${\tilde{P}}_{si} (Γ_{k})$ represents the possibility that mixed event $Γ_{k}$ happens to $X_{si}$ , that is, fuzzy probability. Then, ${\tilde{P}}_{si} (Γ_{1}), {\tilde{P}}_{si} (Γ_{2}), \dots, {\tilde{P}}_{si} (Γ_{C})$ are fuzzy state probabilities of attribute node $X_{si}$ . The computation equation is

\begin{matrix} {\tilde{P}}_{si} (Γ_{k}) & = E [{μ'}_{sij} (Γ_{k})] = \sum_{k = 1}^{C} P (x_{sij} \in Γ_{k}) \cdot {μ'}_{sij} (Γ_{k}) \\ \approx \frac{1}{n_{si}} \sum_{k = 1}^{C} n_{sk} \cdot {μ'}_{sij} (Γ_{k}) = \frac{1}{n_{si}} \sum_{j = 1}^{n_{si}} {μ'}_{sij} (Γ_{k}) \end{matrix}

(1)

where $x_{sij} \in Γ_{k}$ means that the $j th$ observed value of the $i th$ reliability factor in system $u_{s}$ belongs to fuzzy subset $Γ_{k}$ , or that mixed event $Γ_{k}$ happens to observed value $x_{sij}$ . $n_{si}$ is observation times. $n_{sk}$ denotes the number of $x_{sij}$ belonging to fuzzy subset $Γ_{k}$ . $μ'_{sij} (Γ_{k})$ is the fuzzy measure of $x_{sij}$ belonging to fuzzy subset $Γ_{k}$ , also known as the Membership Degree. The absolute value of $μ'_{sij} (Γ_{k})$ cannot be known because of its uncertainty, and its relative value is easy to be determined. As can be seen from equation (1), when $μ'_{sij} (Γ_{k})$ only equals to 1 or 0, the model is just the simple Bayesian network. So the simple Bayesian network is a special form of the improved fuzzy Bayesian network, and the improved fuzzy Bayesian network is the general form of the simple Bayesian network.

In order to introduce the improved fuzzy C-means algorithm conveniently, we suppose $n_{si}$ equals to 1, that is, attribute node $X_{si}$ of system $u_{s}$ has only one observed value, and we regard each system as a whole sample. Suppose $m_{k}$ is the center vector of classification $Γ_{k}$ , and $m_{k} = (m_{k 1}, m_{k 2}, \dots, m_{kd}) (k = 1, 2, \dots, C)$ . $\bar{m}$ is the centroid of the particle set which is composed of $C$ classification centers, and $\bar{m} = (1 / C) \sum_{k = 1}^{C} m_{k} = ({\bar{m}}_{1}, {\bar{m}}_{2}, \dots, {\bar{m}}_{d})$ . Obviously, it is a certainty classification. When we regard $m_{k}$ as a sample in $Γ_{k}$ approximately and regard a certain distance from $u_{s}$ to $m_{k}$ as a kind of similarity between $u_{s}$ and $Γ_{k}$ , the certainty of the classification above is changed to uncertainty, and the uncertainty classification is closer to the real classification situation.

Let

‖ u_{s} - m_{k} ‖^{2} = \sum_{y = 1}^{d} w_{y} \cdot (X_{sy} - m_{ky})^{2}

(2)

Equation (2) is a certain distance from $u_{s}$ to $m_{k}$ . It is a weighted distance, and $w_{y} = σ_{y}^{2} / \sum_{z = 1}^{d} σ_{z}^{2}$ , which is a kind of clustering weight. $w_{y}$ represents the proportion of clustering contribution of $X_{sy}$ to that of all attributes. $σ_{z}^{2}$ is the quasi-variance, and $σ_{z}^{2} = \sum_{k = 1}^{C} {(m_{kz} - {\bar{m}}_{z})}^{2} / C$ . Let

μ_{s} (Γ_{k}) = \frac{1}{‖ u_{s} - m_{k} ‖^{2} + ε} / \sum_{l = 1}^{C} \frac{1}{‖ u_{s} - m_{l} ‖^{2} + ε}

(3)

where $ε$ is a constant. It can adjust its excessive influence on the Membership Degree when $| | u_{s} - m_{k} | |^{2}$ is excessively small. Equation (3) is the improved fuzzy Membership function, also known as the Membership Degree. It strictly satisfies three measurement criteria: non-negative boundedness, additivity, and normalization. It is obviously scientific and reasonable to regard it as a Membership function of the unascertained classification.

The general form of the objective function of the improved fuzzy C-means is

J = \sum_{k = 1}^{C} \sum_{s = 1}^{Q} ‖ u_{s} - m_{k} ‖^{2} = \sum_{k = 1}^{C} \sum_{s = 1}^{Q} \sum_{y = 1}^{d} w_{y} \cdot (X_{sy} - m_{ky})^{2}

(4)

Then we can transform the improved fuzzy C-means into the following optimization problem: solve the minimum for $J$ . When $J$ is minimum, the corresponding value of $μ_{s} (Γ_{k})$ is optimum.

If we regard system $u_{s}$ as a sample and all systems as a sample set to compute the Membership Degree with the improved fuzzy C-means algorithm, the above-mentioned $μ_{s} (Γ_{k})$ is a comprehensive Membership Degree corresponding to all the indicators of system $u_{s}$ . In this article, we regard each observed value $x_{sij}$ of attribute node $X_{si}$ as a sample and all the observed values of $X_{si}$ as a sample set, and compute the value of Membership Degree $μ'_{sij} (Γ_{k})$ corresponding to the single indicator $X_{si}$ with the improved fuzzy C-means algorithm. We compute the fuzzy probability of attribute node $X_{si}$ by substituting $μ'_{sij} (Γ_{k})$ into equation (1). Then we compute the average value of $X_{si}$ , and ${\bar{x}}_{si} = (1 / n_{si}) \sum_{j = 1}^{n_{si}} x_{sij} (s = 1, 2, \dots, Q; i = 1, 2, \dots, d)$ . After substituting ${\bar{x}}_{si} (s = 1, 2, \dots, Q; i = 1, 2, \dots, d)$ into the improved fuzzy C-means algorithm, we compute the clustering weight $w_{i}$ of influencing factor $X_{i}$ of system reliability, whose value indicates the importance degree of influencing factor and evaluation indicator $X_{i}$ .

Evaluation experiments

Data acquisition

In this article, one part of the raw data come from the 1-year maintenance records and local weathers of six dynamic face recognition systems in six public security bureaus. Because these data are rather sensitive, we use $u_{1}, u_{2}, \dots, u_{6}$ to represent these six systems, respectively. After analyzing the maintenance records, we obtain relevant basic data, such as failure time, failure location, failure module, failure cause, number of image channels involved, and duration of failure. The other part of the raw data come from the performance evaluation of the six systems on the spot. In this article, we only evaluate the recognition performance of the six systems. These data are used as the basic data of the evaluation experiments.

Modeling

The first task of improved fuzzy Bayesian network modeling is to set the network nodes. We regard all specific influencing factors of reliability as the root nodes of the improved fuzzy Bayesian network, subsystems as intermediate nodes, and the system reliability as the leaf node. The summary of the specific variables serving as the nodes of the model is shown in Table 1. On the basis of analyzing causal relationships between these nodes and basic data, we create an adjacency matrix of the network nodes. We first construct a framework of the improved fuzzy Bayesian network. Then we extend it along the time axis to obtain the improved Fuzzy Dynamic Bayesian Network with an interval of 1 month. It consists of five time slices, namely, five snapshots of the fuzzy Bayesian network. Figure 1 shows a framework of two time slices. For each time slice, there are 21 root nodes (influencing factors of reliability), 6 intermediate nodes, and 1 leaf node. There are other model parameters as follows: the number of systems $Q = 6$ , factor $X_{i} (T) (8 \leq i \leq 28)$ , and the number of node states or classifications $C = 3$ . The three states of attribute nodes are “High,”“Middle,” and “Low,” which are represented by $Γ_{1}, Γ_{2}, and Γ_{3}$ , respectively. $n_{si}$ is observation times of factor $X_{si}$ of system $u_{s}$ , and $1 \leq n_{si} \leq 31$ . The conditional probabilities between parent and child nodes are determined by the following equations

P (X_{c} \in Γ_{1} | \exists X_{p} \in Γ_{1}) = 1

(5)

P (X_{c} \in Γ_{k} | \exists X_{p} \in Γ_{1}) = 0 (k = 2, 3)

(6)

P (X_{c} \in Γ_{2} | \forall X_{p} \notin Γ_{1}, \exists X_{p} \in Γ_{2}) = 1

(7)

P (X_{c} \in Γ_{k} | \forall X_{p} \notin Γ_{1}, \exists X_{p} \in Γ_{2}) = 0 (k = 1, 3)

(8)

P (X_{c} \in Γ_{3} | \forall X_{p} \in Γ_{3}) = 1

(9)

P (X_{c} \in Γ_{k} | \forall X_{p} \in Γ_{3}) = 1 (k = 1, 2)

(10)

where $X_{c}$ represents a child node, $X_{p}$ represents a parent node, “1” means the occurrence of mixed event $Γ_{k}$ , and “0” means that the mixed event $Γ_{k}$ does not occur.

Table 1.

Variables used to construct the improved fuzzy Bayesian network.

Variable	Content
$X_{1}$	Reliability of system
$X_{2}$	Acquisition subsystem
$X_{3}$	Transmission subsystem
$X_{4}$	Storage subsystem
$X_{5}$	Recognition subsystem
$X_{6}$	Display subsystem
$X_{7}$	Control subsystem
$X_{8}$	Power supply for front-end equipment
$X_{9}$	Camera
$X_{10}$	Light supplement lamp
$X_{11}$	Lens
$X_{12}$	Power supply for server room
$X_{13}$	Power supply for transmission line
$X_{14}$	Line
$X_{15}$	Electromagnetic shielding
$X_{16}$	Hardware of storage
$X_{17}$	Software of storage
$X_{18}$	Hardware of identification
$X_{19}$	Software of identification
$X_{20}$	Recognition performance
$X_{21}$	Power supply for display
$X_{22}$	Hardware of display
$X_{23}$	Software of display
$X_{24}$	Power supply for control
$X_{25}$	Hardware of control
$X_{26}$	Software of control
$X_{27}$	Weather
$X_{28}$	Adversary attack

Figure 1.

Improved Fuzzy Dynamic Bayesian Network model. This is a model framework of two time slices, Time $T$ and Time $T + 1$ . $X_{1}$ is the leaf node and query variable. $X_{2}, X_{3}, \dots, X_{7}$ are the intermediate nodes and hidden variables. $X_{8}, X_{9}, \dots, X_{28}$ are the root nodes and state variables. The initial state probabilities of each variable are all set to 0.333.

The prior probabilities of the root nodes can be computed or evaluated through the following procedure. To begin with, we need to set some intermediate parameters. These parameters are some statistical results related to reliability factor failure coefficient $x_{sij}$ on the $j th$ day, including failure rate $λ_{sij}$ , the $t th$ failure restoration time $T_{sijt}^{R}$ , and the number of image channels involved in the $t th$ fault, $n_{sijt}$ . Then the calculation equation of the cumulative failure restoration time on the $j th$ day is

T_{sij}^{AR} = \sum_{t = 1}^{λ_{sij}} n_{sijt} \cdot T_{sijt}^{R}

(11)

When an influencing factor of the reliability is narrow-sense, the calculation equation of its failure coefficient $x_{sij}$ is

\begin{matrix} x_{sij} = \frac{T_{sij}^{AR}}{24 \times 60 \times V_{s}} \\ (s = 1, \dots, 6; i = 8, \dots, 28; i \neq 20; j = 1, \dots, n_{si}) \end{matrix}

(12)

where $V_{s}$ is the total number of image channels in system $u_{s}$ .

When an influencing factor of the reliability is broad-sense, to be specific, the recognition performance is different from the narrow-sense factors in calculation method, whose calculation equation is

x_{sij} = 1 - R_{s 1} R_{s 2} R_{s 3} (s = 1, \dots, 6; i = 20; j = 1)

(13)

where $R_{s 1}$ is the face capture rate, $R_{s 2}$ is the face qualification rate, and $R_{s 3}$ is the True Positive Identification Rate $(TPI R_{s})$ of system $u_{s}$ at 1% False Positive Identification Rate $(FPI R_{s})$ in $M : N$ mode, abbreviated as $TPI R_{s}$ at 1% $FPI R_{s}$ . Because of the stability of the recognition performance and the inconvenience of the repetitive evaluation experiments on the spot, we evaluate and compute the recognition performance only once for each system in each month.

The specific evaluation process of $R_{s 1}$ is as follows: Randomly sample $N_{s 1}$ 5-min clips captured by $N_{s 1}$ face cameras of system $u_{s}$ in a certain proportion. Use a video-based pedestrian statistics algorithm to count the number of pedestrians in the $i th$ clip, and let it be $A_{s 1 i}$ . Remove repetitive faces captured by a same camera in the corresponding period and get the actual number of faces captured by the $i th$ camera, and let it be $A_{s 2 i}$ . Then, $R_{s 1} = (\sum_{i = 1}^{N_{s 1}} A_{s 2 i} / A_{s 1 i}) / N_{s 1}$ .

The specific evaluation process of $R_{s 2}$ is as follows: Randomly sample 15,000 unblocked, clear, and frontal face images from the six systems in a certain proportion and give them a positive label. We call them qualified face images, namely, positive samples. Randomly sample 15,000 blocked, blurred, or non-frontal face images from the six systems in a certain proportion and give them a negative label. We call them unqualified face images, namely, negative samples. Design a convolution neural network model and train it with the above-mentioned labeled sample data. Apply the trained convolution neural network model to classify the face images captured by each of the above-mentioned cameras into two classifications, qualified and unqualified. Count the number of qualified face images captured by the $i th$ camera, and let it be $A_{s 3 i}$ . Then, $R_{s 2} = (\sum_{i = 1}^{N_{s 1}} A_{s 3 i} / A_{s 2 i}) / N_{s 1}$ .

The specific evaluation process of $R_{s 3}$ is as follows: Randomly sample 1000 pairs of face images from LFW, LFW BLUFR, MegaFace, and a self-built face dataset, respectively. Each pair of faces belong to the same identity. Divide all sample pairs into two face sets, each of which contains 4000 face images. One serves as a positive probe set and the other as a gallery set. Randomly sample 500 pairs of face images from the four above-mentioned face datasets, respectively, which serve as a negative probe set. The negative probe set has no overlap with the positive probe set, and each pair of faces belong to different identities. Conduct an evaluation experiment with the three face sets on the platform of system $u_{s}$ to get the number of True Positive outcomes $(T P_{s})$ and that of False Negative outcomes $(F N_{s})$ . At this time, $M = 6000, N = 4000$ . Then $R_{s 3}$ , that is, $TPI R_{s}$ at 1% $FPI R_{s}$ , is reported by the evaluation experiment, whose computation equation is $R_{s 3} = TPI R_{s} = T P_{s} / (T P_{s} + F N_{s})$ . The images of the self-built face dataset are selected from the six dynamic face recognition systems in a certain proportion.

According to the series of experiments and calculations above, we obtain the reliability factor failure coefficients and some related parameters. For example, a batch of statistical results of system $u_{5}$ are shown in Table 2. The reliability factor failure coefficients serve as the observed values of the root nodes in the improved Fuzzy Dynamic Bayesian Network. As for root node $X_{i}$ , we serve all the observed values of six systems in 1 year as one sample set and put them into the improved fuzzy C-means algorithm to obtain single indicator Membership Degree $μ'_{sij} (Γ_{k})$ of $x_{sij}$ on each observation day. When we place each single indicator Membership Degree of each system into equation (1) on a monthly basis, we obtain the monthly fuzzy probabilities and the classification of $X_{si}$ . Then we serve the fuzzy probabilities of $X_{si}$ in the 12th month as the prior probabilities of the root nodes in the improved Fuzzy Dynamic Bayesian Network model. For each pair of adjacent nodes in two time slices, the state transition probabilities are determined by the statistics on monthly classification changes. One of the state transition matrices is shown in Table 3. So far, the modeling tasks of the improved Fuzzy Dynamic Bayesian Network are completed.

Table 2.

A batch of statistics on reliability factor failure coefficient and related parameters.

Factor	Equivalent failure rate (times)	Cumulative failure restoration time (min)	Failure coefficient
$X_{8}$	0	0	0
$X_{9}$	0	0	0
$X_{10}$	0	0	0
$X_{11}$	0	0	0
$X_{12}$	60	948	0.011
$X_{13}$	0	0	0
$X_{14}$	0	0	0
$X_{15}$	0	0	0
$X_{16}$	0	0	0
$X_{17}$	0	0	0
$X_{18}$	0	0	0
$X_{19}$	0	0	0
$X_{20}$	-	-	0.177
$X_{21}$	0	0	0
$X_{22}$	0	0	0
$X_{23}$	0	0	0
$X_{24}$	0	0	0
$X_{25}$	0	0	0
$X_{26}$	0	0	0
$X_{27}$	60	10,800	0.125
$X_{28}$	0	0	0

The data in this table are the statistical results based on a performance evaluation report and a day’s maintenance records of system $u_{5}$ , as well as the local weather severity of the day.

Table 3.

State transition matrix of $X_{8}$ in system $u_{1}$ .

$X_{81} (T)$	$X_{81} (T + 1)$
	High	Middle	Low
High	88.7	10.4	0.9
Middle	62.9	33.1	4.0
Low	40.0	43.9	16.1

Inference and solving

In this article, a piece of software named Netica is used for inference and solving. First, the model is initialized. The initial state parameters of each node are all set to 0.333. When the fuzzy prior probabilities of root nodes are put into the network, root node states of the network are updated. As soon as the conditional probabilities and the state transition probabilities are put into the network, the network inference is triggered. Then the fuzzy probability distributions of all nodes are obtained, and the fuzzy reliability state probabilities of a system in each time slice are obtained. As far as system $u_{1}$ is concerned, the sketch of inference and solving is shown in Figure 2. As can be seen in Figure 2, at time T the fuzzy reliability state probabilities of system $u_{1}$ are 0.014, 0.104, and 0.882, which represent the possibilities that system $u_{1}$ has “High,”“Middle,” and “Low” reliability states, respectively. The summary evaluation results of all the six systems are shown in Table 4. The clustering weights of the 21 indicators are shown in Table 5.

Figure 2.

Sketch of inference and solving of the improved Fuzzy Dynamic Bayesian Network model, taking system $u_{1}$ as an example.

Table 4.

Fuzzy reliability state probabilities.

System	Reliability state (High, Middle, Low)
	$T$	$T + 1$	$T + 2$	$T + 3$	$T + 4$
$u_{1}$	0.882	0.841	0.834	0.831	0.831
	0.104	0.142	0.148	0.150	0.151
	0.014	0.017	0.018	0.019	0.019
$u_{2}$	0.737	0.729	0.730	0.731	0.731
	0.236	0.248	0.247	0.247	0.247
	0.027	0.023	0.023	0.023	0.023
$u_{3}$	0.563	0.438	0.403	0.393	0.390
	0.401	0.463	0.485	0.492	0.494
	0.037	0.099	0.112	0.115	0.116
$u_{4}$	0.455	0.475	0.476	0.476	0.476
	0.505	0.396	0.394	0.394	0.394
	0.039	0.129	0.130	0.130	0.130
$u_{5}$	0.536	0.431	0.414	0.411	0.411
	0.426	0.468	0.474	0.476	0.476
	0.038	0.101	0.111	0.113	0.113
$u_{6}$	0.715	0.749	0.758	0.760	0.760
	0.257	0.227	0.220	0.219	0.218
	0.028	0.024	0.022	0.022	0.021

Bold is the maximum state probability of a system in each time slice.

Table 5.

Clustering weights of indicators.

Indicator	$X_{8}$	$X_{9}$	$X_{10}$	$X_{11}$	$X_{12}$	$X_{13}$	$X_{14}$	$X_{15}$	$X_{16}$	$X_{17}$	$X_{18}$
Weight	0.000207	0.006898	0.005300	0.006945	0.294319	0.000175	0.005575	0.000001	0.000048	0.006351	0.000076
Indicator	$X_{19}$	$X_{20}$	$X_{21}$	$X_{22}$	$X_{23}$	$X_{24}$	$X_{25}$	$X_{26}$	$X_{27}$	$X_{28}$
Weight	0.003653	0.670027	0.000203	0.000000	0.000021	0.000070	0.000033	0.000000	0.000098	0.000000

The higher the weight, the more important an indicator.

Analysis and verification of results

As shown in Table 4, at time $T + 4$ , the “High” reliability state probabilities of $u_{1}, u_{2}, u_{4}, and u_{6}$ are higher than those of the other two states correspondingly. The “High” reliability state probability of $u_{1}$ is the highest. According to the maximum probability identification criterion, we determine that the reliability states of $u_{1}, u_{2}, u_{4}, and u_{6}$ are all “High.” Similarly, the “Middle” reliability state probabilities of $u_{3}$ and $u_{5}$ are higher than those of the other two states correspondingly. According to the maximum probability identification criterion, we determine that the reliability states of $u_{3}$ and $u_{5}$ are all “Middle.”

As can be seen in Table 5, there are two indicators with far larger clustering weights, $X_{20}$ (recognition performance) and $X_{12}$ (power supply for server room). The percentages of the two indicators in the whole index system are 67.0027% and 29.4319%, respectively, which means that the dispersion degrees of the two reliability factor failure coefficients are relatively high in all the six systems. The two indicators make greater contributions to the evaluation results. By comparing the two failure coefficients of the six systems with each other, we find that those of system $u_{1}$ are the lowest in the six systems, while those of $u_{5}$ are the highest in the six systems. The primary reason for this is that system $u_{1}$ belongs to a subway police substation in a city of south China. The face cameras of system $u_{1}$ are distributed on subway pavements inside. The decent conditions of system $u_{1}$ , such as high pedestrian flow volume, moderate illumination, superior algorithm, and reliable power, make it the highest reliability, while system $u_{5}$ is deployed in a northern coastal city of China. The unfavorable conditions of system $u_{5}$ , such as inappropriate camera layouts, much bad weather, unreliable power, and inferior algorithm, make it the lowest reliability. This partly verifies that the evaluation results are realistic.

In addition, we verify the evaluation results according to three kinds of actual achievements of the systems, including arresting criminal suspects, breaking cases, and warning to key population. First, we use $FPI R'_{i}$ at 1% $FNI R'_{i} (i = 1, 2, 3)$ to represent the $i th$ actual achievements of a system. Then we serve $FPI R'_{1}$ , $FPI R'_{2}$ , and $FPI R'_{3}$ as three root nodes and the total actual achievements as a leaf node to construct a framework of the improved Fuzzy Dynamic Bayesian Network. The computation process of $FPI R'_{i}$ is as follows: Suppose the probability that the target faces appear in the camera fields of view of the system equals 1. Regard all face images captured by the system in the evaluation period as a probe set whose size is $M'$ and the $i th$ kind of watch targets in the evaluation period as a gallery set whose size is $N'_{i}$ . Let $T P'_{i}$ be correct warning times and $F P'_{i}$ be false alarm times in the evaluation period. Transform $FPI R'_{i} = F P'_{i} / (F P'_{i} + T N'_{i})$ into $T N'_{i} = FPI R'_{i} \cdot F P'_{i} / (1 - FPI R'_{i})$ . Then

\begin{matrix} FPI R'_{i} & = F P'_{i} / (T {P'}_{i} + F N'_{i}) \\ = (M' - {N'}_{i} - \frac{FPI {R'}_{i}}{1 - FPI {R'}_{i}} F {P'}_{i}) / \\ (T {P'}_{i} + M' - {N'}_{i} - \frac{FPI {R'}_{i}}{1 - FPI {R'}_{i}} F {P'}_{i}) \end{matrix}

(14)

We substitute the known parameters of system $u_{s}$ into equation (14) to obtain the value of its $FPI R'_{i}$ . We serve all the values of $1 - FPI R'_{i}$ of the six systems in 1 year as the observed values of the root nodes in the improved Fuzzy Dynamic Bayesian Network. We compute the actual achievement state probabilities in the same way as in the previous reliability evaluation, which are shown in Table 6. We compare the evaluation results of each system with its actual achievements and then find that the reliability state probabilities and the corresponding actual achievement state probabilities are very close to each other, and the reliability of each system is consistent with its actual achievements, that is, the evaluation results are correct and reasonable. The comparisons of the evaluation results of $u_{1}$ and $u_{5}$ with their actual achievements are shown in Figure 3.

Table 6.

Actual achievement state probabilities.

System	Actual achievement state (High, Middle, Low)
	$T$	$T + 1$	$T + 2$	$T + 3$	$T + 4$
$u_{1}$	0.883	0.839	0.830	0.828	0.828
	0.101	0.149	0.158	0.160	0.160
	0.016	0.012	0.012	0.012	0.012
$u_{2}$	0.736	0.731	0.734	0.736	0.737
	0.248	0.232	0.226	0.224	0.223
	0.016	0.038	0.040	0.041	0.040
$u_{3}$	0.560	0.437	0.403	0.393	0.389
	0.409	0.467	0.481	0.485	0.486
	0.031	0.095	0.116	0.122	0.124
$u_{4}$	0.459	0.479	0.478	0.478	0.477
	0.510	0.395	0.394	0.394	0.394
	0.031	0.126	0.128	0.129	0.129
$u_{5}$	0.538	0.431	0.413	0.409	0.408
	0.431	0.464	0.466	0.466	0.466
	0.031	0.105	0.122	0.125	0.126
$u_{6}$	0.720	0.751	0.754	0.754	0.754
	0.259	0.226	0.224	0.223	0.223
	0.021	0.023	0.022	0.022	0.022

Bold is the maximum state probability of a system in each time slice.

Figure 3.

Comparisons of evaluation results with actual achievements, taking $u_{1}$ and $u_{5}$ as examples: (a) “High” state probabilities of the reliability and actual achievements, (b) “Middle” state probabilities of the reliability and actual achievements, and (c) “Low” state probabilities of the reliability and actual achievements.

The original intention and the primary purpose of the reliability evaluation are to find vulnerabilities in the light of the evaluation results and improve the reliability effectively. According to Table 5, we know that factor $X_{20}$ exerts the greatest influence on the reliability, and factor $X_{12}$ comes second. If the reliability of a system is low, the two indicators are the cruxes of the matter. It is of great significance to decrease failure coefficients of the two indicators for the reliability improvement of the system. As far as system $u_{5}$ is concerned, the failure coefficients of $X_{20}$ and $X_{12}$ are relatively high. So the public security bureau should improve the reliability from the two aspects. First, improve the system recognition performance. In detail, optimize the location layouts and deployment parameters of front-end cameras, and improve or upgrade the face recognition algorithm constantly. Then reduce power failures of the system in server room. Specifically, equip the server room with enough generators or expand capacity for unattended power source (UPS).

Conclusion

In this article, we propose a novel evaluation method with TPIR in dynamic and $M : N$ mode, create a novel reliability evaluation model of face recognition systems in unconstrained scenes, and conduct evaluation experiments on six dynamic face recognition systems in practical use. Then we use Netica to infer and solve the fuzzy reliability state probabilities of the six systems and find out two most important influencing factors. By comparing the evaluation results with the actual achievements of the six systems, we prove that the evaluation method is scientific and reasonable, and the evaluation results are objective and correct. Finally, we put forward two strategies for improving the reliability of a system according to the evaluation results. The reliability promotion strategies based on the evaluation results are strongly well-directed and operable. The improved Fuzzy Dynamic Bayesian Network model proposed in this article is applicable for evaluating the reliability of complex systems characterized by fuzziness, randomness, and continuity. The pioneered reliability evaluation method can contribute to the advancement of face recognition performance evaluation and system reliability analysis. At the same time, it is conducive to improving the effectiveness of dynamic face recognition systems.

Footnotes

Handling Editor: Wei-Chang Yeh

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Basic Special Project of Ministry of Public Security of China (grant number 2016GABJC01) and the National Key R&D Program of China (grant number 2016YFC0801003).

ORCID iD

Zhiqiang Liu

References

Zhang

ZR.

Talking about the construction and management of video surveillance system in public security. China Public Sc 2013; 13: 130–133.

James

Gandhi

Deshmukh

SG.

Fault diagnosis of automobile systems using fault tree based on digraph modeling. Int J Syst Ass Eng Manage 2018; 9(2): 494–508.

Yan

Gao

, et al. Simplified Markov model for reliability analysis of phased-mission system using states merging method. J Shanghai Jiaotong Univ (Sci) 2018; 23: 1–5.

Patowary

Hazarika

Sriwastav

GL.

Reliability estimation of multi-component cascade system through Monte-Carlo simulation. Int J Syst Ass Eng Manage 2018; 9(6): 1–8.

Xue

Liu

, et al. Rigid-flexible coupling virtual prototyping-based approach to the failure mode, effects, and criticality analysis. Int J Adv Manuf Tech 2019; 100(5–8): 1695–1717.

Park

Ahn

Assessment of structural risks using the fuzzy weighted Euclidean FMEA and block diagram analysis. Int J Adv Manuf Tech 2018; 99(9–12): 2071–2080.

Qian

, et al. Research on operational reliability of digital control device of nuclear pressurizer based on dynamic fault tree. In: Li

Zhang

Chen

, et al. (eds) International conference on intelligent manufacturing and Internet of Things. Singapore: Springer, 2018, pp.46–57.

Research on methods of reliability analysis for integrated supervision & control system in metro. PhD Thesis, Southwest Jiaotong University, Chengdu, China, 2011.

Sheng

Ren

Wan

, et al. Reliability analysis of aero-engine main fuel system based on GO methodology. J Shanghai Jiaotong Univ (Sci) 2018; 23(6): 770–775.

10.

Offutt

Thummala

Testing concurrent user behavior of synchronous web applications with Petri nets. Softw Syst Model 2019; 18(2): 913–936.

11.

Wen

Yang

, et al. Reliability evaluation of compressor systems based on universal generating function method. J Shanghai Jiaotong Univ (Sci) 2018; 23(2): 291–296.

12.

Guo

Huang

, et al. Characterizing wafer stage transmission errors via binary decision diagram and dynamic fault tree. J Mech Sci Technol 2018; 32(11): 5111–5119.

13.

Kabir

Papadopoulos

Applications of Bayesian networks and Petri nets in safety, reliability, and risk assessments: a review. Safety Sci 2019; 115: 154–175.

14.

Cai

Kong

Liu

, et al. Application of Bayesian networks in reliability evaluation. IEEE T Ind Inform 2019; 15(4): 2146–2157.

15.

Khakzad

Landucci

Reniers

Application of dynamic Bayesian network to performance assessment of fire protection systems during domino effects. Reliab Eng Syst Safe 2017; 167: 232–247.

16.

Heng

Zheng

Kaewunruen

, et al. Dynamic Bayesian network-based system-level evaluation on fatigue reliability of orthotropic steel decks. Eng Fail Anal 2019; 105: 1212–1228.

17.

Zhu

Zhang

Fatigue damage assessment of orthotropic steel deck using dynamic Bayesian networks. Int J Fatigue 2019; 118: 44–53.

18.

Lee

Choi

DS.

Analysis of the reliability of a starter-generator using a dynamic Bayesian network. Reliab Eng Syst Safe 2020; 195: 106628.

19.

HT.

The research on key technologies of effectiveness evaluation for security system. PhD Thesis, Wuhan University, Wuhan, China, 2014.

20.

Wang

DD.

Failure risk assessment of video surveillance system based on fuzzy analytic hierarchy process. J Shenyang Univ 2016; 28(2): 128–131.

21.

Jousselme

Maupin

Bosse

Uncertainty in a situation analysis perspective. In: Proceedings of the sixth international conference of information fusion Australia, 2003, Cairns, QLD, Australia, 8–11 July 2003, pp.1207–1214. New York: IEEE.

22.

HC.

Fuzzy Bayesian system reliability assessment based on exponential distribution. Appl Math Model 2006; 30(6): 509–530.

23.

Penz

Flesch

Nassar

, et al. Fuzzy-Bayesian network for refrigeration compressor performance prediction and test time reduction. Expert Syst Appl 2012; 39(4): 4268–4273.

24.

Shan

Liu

Sun

PL.

Risk analysis on leakage failure of natural gas pipelines by Fuzzy Bayesian Network with a Bow-Tie model. Sci Programming 2017; 2017: 3639524.

25.

Xue

Wang

XF.

Fault diagnosis of multi-state gas monitoring network based on fuzzy Bayesian net. Pers Ubiquit Comput 2019; 23(3–4): 573–581.

26.

Das

Ghosh

SK.

FB-STEP: a fuzzy Bayesian network based data-driven framework for spatio-temporal prediction of climatological time series data. Expert Syst Appl 2019; 117: 211–227.

27.

Islam

Nepal

A fuzzy-Bayesian model for risk assessment in power plant projects. Procedia Comput Sci 2016; 100: 963–970.

28.

Wang

Shan

Risk assessment of mine ignition sources using fuzzy Bayesian network. Process Saf Environ 2019; 125: 297–306.

29.

Zarei

Khakzad

Cozzani

, et al. Safety analysis of process systems using Fuzzy Bayesian Network (FBN). J Loss Prevent Proc 2019; 57: 7–16.

30.

Xia

Foe intention inference in air combat based on fuzzy dynamic Bayesian network. J Hefei Univ Technol 2013; 36(10): 1210–1216.

31.

Wang

Yang

Zhang

, et al. Intuitionistic Fuzzy Dynamic Bayesian Network and its application to terminating situation assessment. Procedia Comput Sci 2019; 154: 238–248.

32.

Chang

Zhang

, et al. Dynamic Bayesian networks based approach for risk analysis of subsea wellhead fatigue failure during service life. Reliab Eng Syst Safe 2019; 188: 454–462.

33.

Kamarujjaman Maitra

. 3D unsupervised modified spatial fuzzy C-means method for segmentation of 3D brain MR image. Pattern Anal Appl 2019; 22(4): 1561–1571.

34.

Mishra

Bhoi

Microarray filtering-based fuzzy C-means clustering and classification in genomic signal processing. Arab J Sci Eng 2019; 44(11): 9381–9395.