FRAMB: A man-machine cooperation risk assessment method for intelligent network information systems

Abstract

The intelligent network information systems, such as smart grid systems, face many security problems in the aspects of sensing, communication and computing. Information security risk assessment is an important way to assess the threats faced by information systems before risk events occur and ensure the security of assets. However, a comprehensive risk assessment of the system is a very resource-consuming process. Many existing risk assessment methods rely on a large number of experts and computing resources. Their assessment results are vulnerable to the differences in experts’ subjective judgments. Therefore, we propose FRAMB, a novel man-machine collaborative risk assessment method based on fitting upper and lower bounds. Firstly, we present a risk assessment criterion including four categories and sixteen risk factors following the ISO/IEC 27005:2018 standard. On this basis, we present the DFAHP and CM-NN assessment models to obtain the upper and lower bounds of the risk assessment value, which provides a reference for expert assessment. FRAMB integrates the experts’ assessment value and the values of upper and lower bounds, and adjusts the weights of these values to give the final risk assessment value. We introduce the risk assessment process of FRAMB in detail through a case study of the smart grid system risk assessment. We evaluate the effectiveness and accuracy of FRAMB through experiments. The experimental results show that FRAMB can effectively and accurately assess the security risks of the intelligent network information systems.

Keywords

Risk assessment information systems neural network analytic hierarchy process expert evaluation

1 Introduction

With the rapidly increasing number of nodes in the intelligent network information systems and the growing complexity of the system functions and architectures, the information systems not only bring great convenience to people’s daily life but also bring more security risks. In 2022, there were more than 9000 newly submitted CVE vulnerabilities on the CVE (common vulnerabilities & Exposures) website [1], of which more than 20% were high-risk vulnerabilities. Once these vulnerabilities are exploited by attackers, the systems may suffer from performance degradation, data leakage, and even the destruction of the whole system. Therefore, it is necessary to assess the risks of information systems to find and solve the potential security problems as soon as possible. However, accurate and comprehensive assessments are often time-consuming and require significant human resources, which is a heavy burden for small and medium-sized enterprises. In this context, how to achieve accurate and reliable risk assessment with limited resources has become a research hotspot in recent years.

Although there have been some researches on system risk assessment, there are still the following problems: 1) The mixed method of qualitative and quantitative assessment requires a large amount of system data and complex computational methods, and may face fuzzy problems without standard answers. 2) The assessment method based on Group Multi-Criteria Decision-Making (GMCDM) needs many experts to participate in the risk assessment of the system to avoid the deviation caused by subjectivity, which has a great demand for personnel and resources. 3) The assessment method based on machine learning needs large-scale assessment benchmark set data as training data, and the models produced by different training processes will produce different assessment results, resulting in the deviation of assessment results.

To solve these problems, we follow ISO/IEC 27005 standard [2] and propose the FRAMB method by combining experts, neural network, and analytic hierarchy process (AHP). FRAMB aims to help experts provide more accurate and reliable analysis and reduce the differential impact of experts’ subjective judgments. This method uses the AHP model and neural network model constructed in this paper to calculate the upper and lower bounds of the system risk assessment value according to the risk assessment criterion proposed in this paper. The upper and lower bounds can be used as the reference interval of the risk assessment value for experts, and can also be directly integrated with the expert assessment value to obtain objective and reliable risk assessment results, according to a certain weight. To make the assessment process conform to the standardized process, most existing risk assessment models follow the assessment process in the ISO/IEC 27005 standards [3, 4]: a) Risk assessment preparation, b) Risk factor assessment, c) Risk determination, d) Risk assessment, e) Risk control. The FRAMB method also follows the above process. Table 1 shows the comparison of the expert assessment method, neural network (NN) assessment method, and FRAMB method under this process.

Table 1
Comparison of three methods in risk assessment process

Process Expert assessment method NN assessment method FRAMB method

a) Risk i. Build an expert team. i. Determine assessment i. Determine assessment criterion.

assessment ii. Develop an assessment criterion. ii. Use data set to train and get the

preparation plan. ii. Use data set to train risk lower bound assessment model.

iii. Determine assessment and get the assessment iii. Construct risk upper bound

criterion and method. model. assessment model.

b) Risk i. Asset analysis. i. Determine risk items i. Analyze overall deployment

factor ii. Threat analysis. according to assessment and functions of system.

assessment criterion. ii. Determine risk items according

c) Risk i. Vulnerability analysis ii. Assess the value of to assessment criterion, and

determination ii. Derive risk severity. risk items. analyze assets, threats and

iii. Obtain system vulnerabilities.

risk characteristics. iii. Assess the value of risk items.

iv. Obtain system risk characteristics.

d) Risk i. Compare with risk i. Input risk characteristics i. Input risk characteristics into

assessment assessment criterion. into the assessment model. the assessment models.

ii. Conduct risk assessment. ii. Get the upper and lower bounds

iii. Determine risk value ii. Obtain risk value of system risk assessment value and

/ risk level. / risk level. the corresponding risk reference

interval.

iii. Get the expert assessment result.

iv. Obtain the weights of upper value,

lower value and expert value.

v. Get the final risk value / risk level.

e) Risk i. Determine whether to accept the risk.

control ii. Give risk control methods and suggestions.

The combination of the expert assessment method and neural network method makes FRAMB have the following advantages: (1) the expert assessment method requires extensive manual review and analysis, which is vulnerable to expert subjectivity. By introducing neural network and AHP models, the FRAMB method can obtain a reasonable range of system risk for experts’ reference. FRAMB can also reduce the impact of experts’ subjectivity. (2) FRAMB does not need a large number of experts and evaluation resources. Enterprises can adjust the number of experts according to the actual situation, so as to reduce the burden on small and medium-sized enterprises. (3) FRAMB method combines neural network, AHP, and expert assessment methods, which can eliminate the deviation of a single model and improve the assessment accuracy. The main contributions of this paper are as follows:

1) We present a risk assessment criterion including four categories and sixteen risk factors. On this basis, we propose a man-machine cooperation risk assessment method FRAMB. FRAMB combines the expert assessment, AHP and neural network methods, which can not only give an objective and reasonable upper and lower bounds of system risk assessment value for experts’ reference but also integrate the upper and lower bound values and expert assessment value to obtain a more objective and accurate system risk assessment value.

2) We present a DFAHP evaluation model, which combines AHP and decision factors to calculate the objective and accurate upper bound of the system risk assessment value. We also present a CM-NN evaluation model that uses a combined BP neural network to give the lower bound of the risk assessment value.

3) A risk assessment example of a smart grid system is given to introduce the risk assessment process of FRAMB in detail. We verify the effectiveness and accuracy of the FRAMB method through experiments. The experimental results show that FRAMB has better accuracy, precision, and recall than the existing methods.

The paper is organized as follows. Section 2 introduces the related work. Section 3 presents the FRAMB framework and risk assessment criterion. Section 4 presents the risk characteristics calculation method, CM-NN model, DFAHP model, and the calculation method of system risk assessment value. The risk assessment process of the FRAMB method is described through an actual risk assessment case in Section 5. The FRAMB method is evaluated in Section 6. Section 7 concludes this paper.

2 Related work

This section introduces three existing risk assessment methods: the mixed methods of qualitative and quantitative assessment [5 –13], the assessment methods based on GMCDM [14 –19] and the assessment method based on machine learning [20 –24].

Some research studies [9 –11] use a combination of qualitative and quantitative methods to assess the risk of the system. Such methods establish risk assessment criteria through qualitative methods and quantitatively calculate the risk value through mathematical models. To overcome the limitations of traditional analysis methods on failure modes and effects, Wen et al. [8] propose a risk assessment method combining subjective and objective weights under uncertainty. Wang et al. [9] propose a network security evaluation model based on AHP and D-S evidence theory, which is combined with the hierarchical model of security evaluation to simplify the security evaluation problem. Zhang et al. [10] establish a mobile commerce privacy security risk assessment and measurement model based on information entropy and Markov chain, which can provide quantitative results s in assessing privacy disclosure risk. Alali et al. [11] use a fuzzy inference model to establish a cyber security risk assessment model based on four risk factors: vulnerability, threat, possibility, and impact. However, the accuracy of the evaluation results of such methods depends much on the knowledge and experience of experts. If the experts are inexperienced, it is easy to lead to inaccurate evaluation results. Moreover, there are often fuzzy problems without "standard answers" in the process of risk assessment, which makes the assessment results of such methods vulnerable to the subjectivity of experts, resulting in the lack of objectivity of the risk assessment results and deviation from the actual situation of the system.

To solve the problem that the risk assessment results are easily affected by the subjectivity of experts, some research studies [16 –19] use the assessment method based on group multi-criteria decision making (GMCDM) to improve the objectivity of the assessment results by increasing the number of experts. In the GMCDM method, how to reasonably set the weights of risk factors and experts is an important issue that affects the decision-making results. Verma et al. [16] propose a thermodynamic consistent GMCDM model using thermodynamic indicators (energy, radiative instinct, and entropy). Aiming to overcome some defects in traditional failure mode and effect analysis (FMEA), Wang et al. [17] combine a variety of multi-criteria decision-making (MCDM) techniques with probabilistic hesitant fuzzy linguistic term sets (PHFLTS) to implement risk assessment of failure modes by a panel of specialists. To explain the uncertainty in evaluation, Esangbedo et al. [18] propose a grey-point-allocation full-consistency (Grey-PA-FUCOM) weighting method and a hybrid MCDM method to evaluate information systems. Considering real constraints of firms such as capital and time, Yener et al. [19] propose a three-stage intuitionistic fuzzy risk assessment method based on FMEA. However, the assessment method based on GMCDM [16 –19] has a large demand for personnel and resources, which is a heavy burden for small and medium-sized enterprises.

The neural network method usually uses data sets to train the assessment model, then obtains the corresponding system risk characteristics according to the assessment criterion, and inputs the risk characteristics into the assessment model to obtain the assessment results. With the development of machine learning technology, many existing studies [3 , 20–24] have introduced machine learning methods into the risk assessment process. Martinez-Garcia et al. [20] propose deep recurrent entropy adaptive model by using deep learning and entropy measures to solve the system reliability problems. Wang et al. [21] propose a back-propagation neural network (BPNN) optimized by a genetic algorithm to evaluate the risk of knowledge fusion in innovation ecosystems. To assess the vulnerability of computer networks, Wang et al. [22] propose a network security risk assessment method based on a Bayesian network attack graph (B-NAG) model. To improve the accuracy and stability of risk assessment, Li et al. [3] propose an evolutionary algorithm that improved cuckoo search (ICS) to pre-train a BPNN. From the perspectives of the host and the network, Hu et al. [23] conduct a multi-dimensional network security risk assessment framework including risk identification and risk calculation stages. Kumar et al. [24] propose a risk assessment method based on a fuzzy neural network (FNN) and a parameter optimization mechanism based on an interaction strategy to optimize FNN parameters. However, the assessment methods based on machine learning are dependent on data sets to obtain accurate assessment models [25], and these methods may face the problem of local minima, which may result in evaluation bias.

3 Design of the FRAMB risk assessment method

3.1 Risk assessment framework of the FRAMB

FRAMB adopts a combination of the neural network, AHP, and experts. Firstly, experts construct the system risk characteristics (detailed in Section 4.1) according to the risk assessment criterion proposed in Section 3.2. Then, the neural network model CM-NN (detailed in Section 4.2) and DFAHP model (detailed in Section 4.3) are established to generate the upper and lower bounds of the system risk assessment value respectively. Referring to the upper and lower bound values, experts check whether the assessment values they give fall within the upper and lower bounds and determine whether human errors have been introduced into the assessment process due to the errors or the subjectivity. Finally, FRAMB calculates the relatively objective system risk value by integrating and adjusting the weights of expert assessment values, and upper and lower bound values according to strategies in Section 4.4. Fig. 1 shows the framework of the FRAMB method. FRAMB includes the following four steps:

(1) Analyzing and assessing the system by experts.

In this step, the expert evaluates the information system and determines the security threats faced by the system and the possible vulnerabilities of the system. The person in charge of the enterprise provides the asset values of various assets in the system. Then, according to the risk assessment criterion introduced in Section 3.2, the expert builds a 16-dimensional risk characteristic Risk by using the calculation method given in Section 4.1.

(2) Calculating the lower bound of risk values using CM-NN.

CM-NN consists of a neural network NN_Rul that focuses on the decision layer and a neural network NN_idx that focuses on the indicator layer. The structures, parameters, and training methods of NN_Rul and NN_idx are shown in Section 4.2. After training the neural networks, CM-NN takes the 16-dimensional system risk characteristic Risk as input to generate an objective and accurate lower bound of the system risk assessment value.

(3) Calculating the upper bound of risk values using DFAHP.

According to the risk assessment criterion, the DFAHP assessment model is divided into indicator layer, decision layer, and assessment layer. The DFAHP assessment model takes the 16-dimensional system risk characteristic Risk as the input to generate an objective and accurate upper bound of the system risk assessment value. The calculation process and method can be seen in Section 4.3.

(4) Calculating the system risk assessment value.

In this step, experts can refer to the upper and lower bound values given by CM-NN and DFAHP in Step2 and Step3 to check and adjust their assessment values. Then, FRAMB calculates the relatively objective system risk value by integrating and adjusting the weights of expert assessment values, and upper and lower bound values.

Fig. 1

The FRAMB risk assessment framework.

3.2 System risk assessment criterion

Referring to ISO/IEC 27001 Information Security Management Standard [26] and ISO/IEC 27005:2018 (section 8.2.3 and Annex D) [2], we present a risk assessment criterion including four categories and sixteen risk factors, as shown in Fig. 2 The proposed risk assessment criterion is divided into three layers: assessment layer, decision layer and indicator layer. At the decision layer, we divide system risks into four categories: management, software and data, network and traffic, and physical equipment. At the indicator layer, management risks include: personnel management (PM), facility management (FM), security awareness (SA), and user operation (UO). Software and data risks include: software vulnerability (SV), software configuration (SC), access control (AC) and data security (DS). Physical equipment risks include: hardware failure (HF), physical environment (PE), the storage medium (SM), and equipment use (EU). Network and traffic risks include: communication transmission (CT), network intrusion (NI), virus prevention (VP) and security audit (ST). Then, the risk characteristics are denoted as Risk =< PM, FM, SA, UO, SV, SC, AC, DS, HF, PE, SM, EU, CT, NI, VP, ST >.

Compared with the traditional criterion rthe four categories at the decision layer of the risk assessment criterion proposed in this paper cover management rsoftware rdata rnetwork and physical equipment rwhich includes all the risks categories that intelligent network information system could face. At the same time rthe relevant risk factors in each category are succinct but sufficient to cover the corresponding security events and more targeted to intelligent network information systems.

Following ISO/IEC 27001 and ISO/IEC 27005 rwe divide the risk assessment values of 16 factors and 4 decision items in the system risk assessment criterion into five levels: very low rlow rmedium rhigh rand very high. The description and value ranges of the five levels are shown in Table 2.

Fig. 2

System risk assessment criterion.

Table 2

The description and the value ranges of risk levels

Risk Level	Value Range	Description
Very high (5)	[0.8,1]	The occurrence frequency is very high, or it is almost inevitable in most cases, or the corresponding assets are very important, or the resulting losses are very serious.
High (4)	[0.6 r0.8)	The occurrence frequency is high ror it may occur in most cases ror the corresponding assets are important ror the resulting losses are serious.
Medium (3)	[0.4 r0.6)	The occurrence frequency is medium ror it may occur in some cases ror the corresponding assets are less important ror cause certain losses.
Low (2)	[0.2 r0.4)	The probability of occurrence is low ror unlikely occur in most cases ror the corresponding assets are less important and the losses are acceptable.
Very low (1)	[0 r0.2)	The probability of occurrence is very low; or the corresponding assets are not important; or little damage is caused.

4 Models and methods in FRAMB

4.1 Risk Characteristics Calculation Method

Each of the 16 risk factors in the assessment criterion proposed in this paper can be expressed as a quadruple as shown in Equation (1):

$R_{itm} = < A, V, T, SE >$ (1)

Where A represents the assets involved in R_itm. Referring to Section 8.2.2 and Annex B of ISO/IEC 27005:2018 2], A includes services, data, software and hardware. V represents the vulnerability involved in R_itm. Referring to section 8.2.5 and Annex D of ISO/IEC 27005:2018, V includes physical environment protection measures, network structure design, software patch, user account, and organization management. T represents the threats involved in R_itm. Referring to Section 8.2.3 and Annex C of ISO/IEC 27005:2018, T includes software and hardware failures, physical environment impact, inadequate management, malicious code, viruses, and network attacks. SE represents the set of security events that are all possible combinations of elements in sets A, V, and T.

For asset a_i involved in security event SE_i, the person in charge of the enterprise gives the asset value V_{a
_i} according to the importance of the asset. Experts analyze and determine threat t_i, and assess threat level L_{t
_i} according to the occurrence frequency of threat t_i in previous security reports and threat statistics released by international organizations. The vulnerability level L_{v
_i} of v_i is determined by experts according to the severity of the damage caused by threat t_i to assets a_i.

Referring to the risk calculation method in Annex E of ISO/IEC 27005:2018 standard, we present the risk value calculation method of risk factors as shown in Equation (2) and Equation (3). In this paper, the occurrence probability of SE_i and the loss caused by SE_i are respectively expressed as $P_{ocr}^{i}$ and $L_{ocr}^{i}$ as shown in Equation (2). The risk value of risk factors Value_{R
_itm} can be calculated by Equation (3).

$\begin{matrix} P_{ocr}^{i} & = \sqrt{L_{t_{i}} \times L_{v_{i}}} \\ L_{ocr}^{i} & = \sqrt{V_{a_{i}} \times L_{v_{i}}} \end{matrix}$ (2)

${Value}_{R_{itm}} = \sum_{i = 1}^{n} \sqrt{P_{ocr}^{i} \times L_{ocr}^{i}} / n$ (3)

As shown in Table 3, we use an example to illustrate the risk value calculation method of risk factor R_itm. It is assumed that R_itm contains three types of security events {SE_i ∣ i ∈ (1, 3)}. For SE₁, the asset value of a₁ is V_{a
₁} and the relevant vulnerability level of v₁ is L_{v
₁}. Also, a₁ face the threat t₁ with threat level L_{t
₁}. Similarly, we also know the asset values, relevant vulnerability levels and corresponding threat levels in SE₂ and SE₃. By using Equation (2), we can calculate the occurrence probability $(P_{ocr}^{i})$ of SE_i and the loss $(L_{ocr}^{i})$ caused by SE_i. Then, according to Equation (3), we can calculate the risk value of R_itm. In this way, FRAMB can calculate the risk values of 16 risk factors in the risk assessment criterion, and then obtain a 16-dimensional system risk characteristics Risk.

Table 3

Evaluation example of a risk factor

Vulnerability level	Threat level	Asset value	$P_{ocr}^{i} \times L_{ocr}^{i}$
L_{v ₁} = 1	L_{t ₁} = 2	V_{a ₁} = 2	$\sqrt{2} \times \sqrt{2} = 2$
L_{v ₂} = 2	L_{t ₂} = 3	V_{a ₂} = 1	$\sqrt{6} \times \sqrt{2} \approx 3.46$
L_{v ₃} = 4	L_{t ₃} = 1	V_{a ₃} = 2	$\sqrt{4} \times \sqrt{8} \approx 5.66$
Risk Value	${Value}_{R_{itm}} = \frac{\sqrt{2} + \sqrt{3.46} + \sqrt{5.66}}{3} \approx 1.88$

4.2 CM-NN: Neural network assessment model based on comparison method

The neural network is nonlinear, which has strong robustness and fault tolerance. Theoretically, with the increase of data sets, its performance will be better. However, this also means that the neural network is sensitive to the training data, and different weight vectors will be obtained during each training, resulting in different predictions [27]. Therefore, we use the combined neural network that can make full use of the data set to establish the CM-NN evaluation model. In this way, CM-NN can not only compensate for the prediction error caused by different weight vectors of a single BP neural network but also help to reduce the dependence on human experts.

The structure of the CM-NN assessment model is shown in Fig. 3. The CM-NN assessment model is divided into four layers: input layer, hidden layer, output layer, and assessment result layer. Also, there are three parts in this model: index-focused neural network (NN_Idx), decision-focused neural network (NN_Rul), and assessment value comparison mechanism. The learning rate determines the speed of weight update. If it is set too high, the result will exceed the optimal value, and if it is set too low, the decline speed will be too slow. The purpose of momentum rate is to avoid shocks during weight updating process. After many tests, the learning rate and momentum rate of NN_Rul and NN_Idx are set to 0.01 and 0.001 respectively, which can ensure the accuracy while fast convergence. The activation function is Sigmoid, and the epochs is 1000. The neural network is trained using the gradient descent and backpropagation methods.

Fig. 3

Structure of the CM-NN assessment model.

To obtain the lower bound of the system risk assessment value, we reduce the data label values of the training set constructed in Section 6.1 by 0.4 to train the CM-NN assessment model. Because after many tests, we found that the evaluation values obey the normal distribution, as shown in Fig. 12(a) to Fig. 12(c) in Section 5.3, and the length of the 90% confidence interval is about 0.8. So, we consider that the downward deviation of the data label value of 0.4 is reasonable as the lower bound of the assessment value. Therefore, the training set with data label values reduced by 0.4 is taken as a new data set for training NN_Idx. Further, to train the NN_Rul, we calculated the mean values of risk factors in four categories to generate a new training set, in which a piece of data includes four features and data labels with deviation.

After NN_Rul and NN_Idx give the assessment values at the output layer, the values are input to the assessment result layer. Then, CM-NN selects the lower risk value as the final assessment value, which is the lower bound of the system risk value.

4.3 DFAHP: Analytic hierarchy process based on decision factors

AHP is a simple and flexible method for quantitative analysis of qualitative problems. It advances gradually from bottom to top according to different layers of assessment criterion, and finally obtains the overall assessed value. The risk assessment criterion in this paper is composed of an indicator layer, a decision layer, and an assessment layer, as shown in Fig. 2. In order to obtain a reasonable upper bound of the assessment value, we construct a DFAHP assessment model that uses a ranking method to enhance the weight of decision factors with higher risk. In this paper, the decision factors of the decision layer are denoted as R^mm, R^sd, R^ep, and R^nw, and the values of them are denoted as m, s, e, and n, which are the mean values of the risk factors at the corresponding indicator layer. Then, the decision factors are sorted according to the mean values from small to large and then denoted as R1, R2, R3, and R4 respectively. The calculation method of the upper bound value at the assessment result layer is shown in Equation (4). For a piece of data to be evaluated, its 16-dimensional feature is considered as risk factors at the index layer. Then DFAHP gets four decision factors by calculating the mean values of the corresponding risk factors. After sorting the decision factors from small to large, DFAHP uses Equation (4) to get the upper bound value.

${Value}_{target} = \frac{\frac{\frac{R 1 + R 2}{2} + R 3}{2} + R 4}{2}$ (4)

An example of the assessment matrix is given in this section to illustrate the rationality of the DFAHP assessment method. According to Table 2, m, s, e, and n are normalized and changed into assessment levels, which can be expressed as RL_m, RL_s, RL_e and RL_n. Also, the decision factors are expressed as decision vectors $R_{1 \times 5}^{mm}$ , $R_{1 \times 5}^{sd}$ , $R_{1 \times 5}^{ep}$ , and $R_{1 \times 5}^{nw}$ , as shown in Equation (5).

$\begin{matrix} R_{1 \times 5}^{mm} & = {\begin{matrix} R_{i}^{mm} = 0, i \in (1, 5) \cap i \neq {RL}_{m} \\ R_{{RL}_{m}}^{mm} = {RL}_{m} \end{matrix} \\ R_{1 \times 5}^{sd} & = {\begin{matrix} R_{j}^{sd} = 0, j \in (1, 5) \cap j \neq {RL}_{s} \\ R_{{RL}_{s}}^{sd} = {RL}_{s} \end{matrix} \\ R_{1 \times 5}^{ep} & = {\begin{matrix} R_{k}^{ep} = 0, k \in (1, 5) \cap k \neq {RL}_{e} \\ R_{{RL}_{e}}^{ep} = {RL}_{e} \end{matrix} \\ R_{1 \times 5}^{nw} & = {\begin{matrix} R_{l}^{nw} = 0, l \in (1, 5) \cap l \neq {RL}_{n} \\ R_{{RL}_{n}}^{nw} = {RL}_{n} \end{matrix} \end{matrix}$ (5)

The decision vectors are sorted according to the assessment levels from low to high and then expressed as R1_1×5, R2_1×5, R3_1×5, R4_1×5, with the condition of $\sum_{i = 1}^{5} R 1_{i} \leq \sum_{j = 1}^{5} R 2_{j} \leq \sum_{k = 1}^{5} R 3_{k} \leq \sum_{l = 1}^{5} R 4_{l}$ . The evaluation matrix of R1 and R2 is denoted as R12, the evaluation matrix of R12 and R3 is denoted as R123, and the evaluation matrix of R123 and R4 is denoted as R1234. The calculation method is shown in Equation (6), where T1 is expanded as a column vector. We define the vector operation Rule₁ as shown in Equation (7), where G is an m-dimensional vector and H is an n-dimensional vector.

$\begin{matrix} R 12 & = {Rule}_{1} (R 1, R 2) \\ R 123 & = {Rule}_{1} (R 12^{T 1}, R 3) \\ R 1234 & = {Rule}_{1} (R 123^{T 1}, R 4) \end{matrix}$ (6)

$\begin{matrix} {Rule}_{1} (G_{1 \times m}, H_{1 \times n}) = D_{m \times n} \\ d_{ij} = {(g_{i} + h_{j}) / 2 | i \in (1, m), j \in (1, n)} \end{matrix}$ (7)

A 5×125 matrix is generated by the above method. Further, we expand the matrix and normalize it into an 25×25 evaluation matrix. The heat map of the evaluation matrix is shown in Fig. 4. The right side of Fig. 4 shows the assessment levels corresponding to different colors. We can see that in the hierarchical distribution of assessment levels in the evaluation matrix, the proportion of assessment results with higher levels is slightly more than the ones with lower levels, as expected. Because the assessment values given by DFAHP, as the upper bound of the system risk assessment values, are greater than most of the assessment values.

Fig. 4

The heat map of the evaluation matrix.

4.4 Reference interval and assessment results

By using DFAHP and CM-NN models, FRAMB can calculate the upper bound and lower bound of the system risk assessment value as a reference interval. Experts can check whether the risk assessment values given by themselves fall within the reference interval to determine whether the assessment process has introduced human errors or subjective problems. The calculation method of the final system risk assessment value is described in detail as follows. The assessment value given by DFAHP is denoted as Value₁. The assessment value given by CM-NN is denoted as Value₂. The expert assessment value is denoted as Value₃. The weight of Value₃ can be expressed as Equation (8), where N_agent is the number of experts. The weights of Value₁ and Value₂ can be expressed as Equation (9).

$W_{{Value}_{3}} = \frac{1 + N_{agent}}{2 (2 + N_{agent})}$ (8) $W_{{Value}_{1}} = W_{{Value}_{2}} = \frac{1 - W_{{Value}_{3}}}{2}$ (9)

In order to reduce costs, some enterprises may only invite one expert. In this case, the weight of the expert assessment value is 1/3. As the number of experts increases according to actual needs, this weight could also be increased but not exceed 1/2, and the expert assessment value could be the mean value of all the expert assessment values. After calculating the weight, FRAMB calculates the weighted sum of Value₁, Value₂ and Value₃ according to Equation (10) to obtain the risk assessment value (Value_risk) of the information system.

${Value}_{risk} = \sum_{j = 1}^{3} W_{{Value}_{j}} \times {Value}_{j}$ (10)

5 Case study

Fig. 5

The structure of the smart grid system.

This section introduces the assessment process of the FRAMB risk assessment method in detail through a risk assessment example of a smart grid system. In recent years, with the rapid growth of power consumption and the urgent need to reduce global carbon emissions, smart grid system has attracted more and more attention. The smart grid system involves a large number of communication devices, which is scattered and leads to the smart grid system facing severe network security threats. Therefore, we select the smart grid system as an example system for risk assessment.

The assessment object in this case study is a smart grid system. As shown in Fig. 5, the smart grid system consists of servers, firewalls, switches, smart controllers, smart adapters, smart meters, and other equipment. The equipment list of the smart grid system is shown in Table 4.

Table 4

The equipment list of the smart grid system

Equipment	Product information	Function and description
Smart controller	Easergy T300	An automatic intelligent terminal, which provides functions such as distribution network management, fault location, isolation and recovery, distributed energy grid connection, and asset management.
Smart adapter	NETGEAR PLP2000	An adapter, which provides network support for mutual communication and information interaction between transmission substations and distribution substations.
Smart meter	DDSY1352	It can calculate the power consumption time-sharing, and meet the power consumption and electricity charge calculation of each power consumption scenario.
Power generation controller / Power transmission controller /Power distribution controller	Schneider Modicon M580	It can remotely control the equipment, and provide functions such as changing the configuration when the equipment is running, obtaining the equipment status, and fault diagnosis.
Real time data server /SQL server /Web server/E-mail server	Dell EMC PowerEdge XE8545	It is a host model that provides hardware support for operating systems, software, and services.
IDS server	Dell Precision 7540	It is a host model that provides hardware support for operating systems, software, and services.
Server operating system	Ubuntu Server 16.04 LTS	It is an operating system, which is convenient for operators to use.
Software	MySQL 5.1.60	Providing database construction, data-base query, etc.
	Apache HTTP Server V2.4.46	Providing HTTP communication, support website deployment, etc.
	Postfix 3.0.5/ Dovecot 2.3.15	Providing mail receiving and sending functions.
	OSSEC-HIDS-2.8.3	Providing system monitoring and control, log monitoring
		and audit, security event management, etc.
Firewall	Cisco Firepower 4120	Providing functions such as threat defense, encryption and threat detection.
Router	TP-LINK AX1800	Providing centralized management, authentication, access of equipment.
Switch	Cisco Business 250	Providing functions such as device connection to the network, network packet filtering and forwarding.

FRAMB aims to provide efficient and low-cost risk assessment for small and medium-sized enterprises with a small number of experts. In this case study, we invited an expert with more than ten years of experience to assess the risk of the smart grid system. The detailed steps for experts to use FRAMB to assess the smart grid system are as follows:

Step 1. Determine the assessment criterion. In this step, the expert decides to use the risk assessment criterion proposed in this paper, which includes four categories and sixteen risk factors, as the assessment criterion for smart grid system risk assessment.

Step 2. Construct CM-NN and DFAHP assessment models. In this step, experts use the methods described in Section 4.2 and Section 4.3 to establish the CM-NN assessment model and DFAHP assessment model respectively.

Step 3. Analyze overall deployment and functions of system. The structure of the smart grid system is shown in Fig. 5, and the main equipment and software in the system are shown in Table 4. The smart grid system mainly provides the following functions: information provision, business acceptance, customer payment, and energy efficiency analysis. Specifically, information provision includes electricity price policy query, branches query, and announcement query. Business acceptance includes launch of new electricity business, change of electricity business, business fault submission, and suggestions or complaints submission. Customer payment includes payment record query, electricity bill query, online recharge, account management. Energy efficiency analysis includes equipment control, power consumption analysis, power supply analysis, and power quality analysis.

Step 4. Determine risk factors according to assessment criterion, and analyze assets, threats and vulnerabilities. According to the method introduced in Section 4.1, the expert sorts out the relevant vulnerabilities, threats, and affected assets of each risk factor of the system, as shown in Table 5.

Table 5

Related vulnerabilities, threats, and assets involved in risk factors

Factors	Vulnerabilities(V)	Threats(T)	Assets(A)
PM	The person in charge and responsibilities	Equipment failure, or unauthorized	The whole system
	of all aspects are not determined	use of equipment
FM	There is no equipment list and asset	Theft of equipment, or Theft of	The whole system
	management system	media data
	There is no special person to manage,	Tampering with hardware or software,	The whole system
	control and protect all kinds of equipment	or equipment malfunction
SA	Lack of safety awareness education for	Error in use, or breach	The whole system
	personnel	of personnel maintainability
	No safety management system has been	Breach of system maintainability	The whole system
	established
UO	User behavior logs and data are not	Abuse of rights, or remote spying	The whole system
	analyzed regularly
SV	CVE-2021-40438: mod_proxy module	Forging server requests, or illegal	Apache HTTP
	failed to properly validate input	data acquisition	Server
	CVE-2021-39275: ap_esape_quotes()	Buffer overflow, or illegal	Apache HTTP
	fails to properly validate input	data acquisition	Server
	CVE-2021-29157: attacker tricks	Forging rights, or remote spying	Dovecot
	OAuth2 authentication into using
	HS256 authentication key
SC	Ubuntu does not prohibit the remote	Illegal access to server permissions,	Relevant servers
	login of root user	or remote spying
	Apache does not set to prohibit	Trojan horse upload, or illegal	Relevant servers
	PHP from running	processing of data
	Apache session event setting	Server resource occupation,	Relevant servers
	too long	or denial of services
AC	Improper allocation of access	Abuse of rights, or illegal processing	Relevant servers
	privileges for some roles	of data
	Unprotected password table	Forging rights, or password leakage	Relevant servers
	Software uncontrolled download	Software malfunction, or counterfeit	Relevant servers
	behavior	software
DS	MySQL runs with root privilege	Privilege raising attack, or illegal	Relevant servers
	processing of data
	MySQL only root user exists	Rights abuse, or illegal data processing	Relevant servers
	The access rights of important files in	Illegal data acquisition	Relevant servers
	the system are not set to 644 or 600
HF	CVE-2021-22792: CPU null pointer	controller / simulator denial of service	Schneider Modicon
	dereference	M580
	CVE-2020-7513: plaintext storage	Eavesdropping, or configuration	Easergy T300
	of sensitive information	data acquisition
PE	Some cables are exposed from pipelines	Failure of telecommunication equipment	Relevant cables
SM	Some storage media are not preserved	Tampering with hardware, or	Relevant equipment
	by special personnel	theft of media
	Some important data are not backed	Data loss and damage	Relevant data
	up redundantly
EU	Some devices and hosts are not subject	Illegal use of equipment, or	Relevant equipment
	to centralized identity authentication	equipment mis-operation	and data
	Some rooms do not have an access	Random access, or equipment	Relevant equipment
	control system	loss and damage
CT	OSPF authentication encryption	information leakage	Relevant Information
	strength is not enough
	There are default NMS strings,	Tampering with hardware	Relevant Information
	public and private
NI	Syscheck does not run regularly	Tampering with document, or	Relevant data
	corruption of data
	Email alarm function is not enabled	breach of personnel availability	Relevant personnel
	Malicious access is not prohibited	equipment malfunction, or	Relevant services
	by limiting IP address	denial of service
VP	Virus database has not been updated	virus infection, or server	Relevant servers
	for a long time	malfunction
ST	The access behavior of Internet	Abuse of rights, or remote spying	Relevant data
	visitors is not recorded
	The audit cycle of security personnel	Abuse of rights, or illegal	Relevant data
	is too long	data acquisition

Step 5. Assess the value of risk items and obtain system risk characteristics. In this step, the expert gives the vulnerability risk level, threat level, and asset value of each risk factor in Table 5, and then gives a list of the security events in the system as shown in Table 6, according to the method introduced in Section 4.1. As described in Section 4.1, security events are expressed as (V, T, A), where V, T, and A represent vulnerability, threat, and asset respectively.

Table 6

The list of the security events

Risk factors	The set of safety events	Risk factors	The set of safety events
PM	{(2,2,4)}	FM	{(1,3,4), (2,2,4)}
SA	{(2,1,4), (2,2,4)}	UO	{(2,2,4)}
SV	{(3,2,3), (1,3,3), (2,2,3)}	SC	{(2,2,2), (3,3,2), (2,1,2)}
AC	{(2,4,2), (3,3,2), (3,3,2)}	DS	{(1,2,2), (2,2,3), (3,1,3)}
HF	{(1,2,3), (2,1,2)}	PE	{(2,1,2)}
SM	{(2,2,2), (1,2,3)}	EU	{(2,2,3), (1,2,3)}
CT	{(2,1,1), (1,2,1)}	NI	{(2,1,2), (2,2,1), (3,2,3)}
VP	{(4,3,3)}	ST	{(2,2,2), (1,2,2)}

According to Table 6 and the risk characteristics calculation method given in Section 4.1, FRAMB can obtain the following risk characteristics: Risk =<2.38, 2.12, 2.19, 2.38, 2.21, 2.13, 2.6, 1.97, 1.62, 1.68, 1.79, 1.74, 1.19, 2.02, 3.46, 1.71 >

Step 6. Input risk characteristics into the assessment models and generate the upper and lower bounds of the system risk assessment value and the corresponding risk reference interval. In this step, FRAMB inputs the characteristics Risk into the CM-NN model to get the lower bound of the system risk assessment value of 1.63, and inputs Risk into the DFAHP model to get the upper bound of the system risk assessment value of 2.26.

Step 7. Experts give risk assessment values referring to the risk reference interval. In this step, the reference interval given by FRAMB is (1.63, 2.26). And the expert takes ISO/IEC:27005 and other standards as the guidance to assess the risk of the system. The risk value given by the expert is 2, which falls within the reference interval.

Step 8. Adjust the weight and get the risk value / risk level. In this step, according to Equation (8) and Equation (9) in section 4.4, the weight of expert assessment value is 0.33, the weight of CM-NN assessment value is 0.33, and the weight of DFAHP assessment value is 0.33. According to Equation (10), the risk value of the system is 1.93. Further, to calculate the risk level of the system, the assessment value of 1.93 under the range (0, 5) needs to be normalized to 0.386 under the range (0, 1). According to Table 2, the system risk level is level 2.

6 Results and discussions

6.1 Dataset construction

To compare FRAMB with existing methods to evaluate its effectiveness and accuracy, we construct the dataset referring to the dataset construction method in [3] and [28]. First, we used the smart grid system mentioned in Section 5 as a prototype system. And we adjust the system configuration parameters included in the 16 risk factors of system risk assessment criterion. For example, we can change the configure of servers to allow remote login or remote root user, which may cause the risk factors (SC) to increase. In this way, we can get many derivative systems. Then, we invite three groups of expert teams to evaluate the systems and give the system risk values. Then, according to Algorithm 1, the final risk value of the system is obtained as a data tag. Finally, we use the risk characteristics calculation method in Section 4.1 to construct the system risk characteristics and insert the data tag to obtain the data with sixteen risk characteristics and one data tag. In this way, we build a data set containing 1000 pieces of data. The structure of the data set is shown in Table 8.

Algorithm 1 Divergence resolution algorithm
Input:EV₁, EV₂, and EV₃
Output: Evalue _fin
Start
1. rank₁, rank₂, rank₃ = round (EV₁), round (EV₂), round (EV₃)
2. if (rank₁ == rank₂ == rank₃):
3. Evalue_fin = average(EV₁, EV₂, EV₃)
4. else if (rank₁ != rank₂ != rank₃):
5. ave = average(EV₁, EV₂, EV₃)
6. m1, m2, m3 are equal to EV₁ - ave, EV₂ - ave, EV₃ - ave
7. w1, w2, w3 are equal to m1, m2, m3 divided by the sum of m1, m2, m3
8. Evalue_fin = the sum of w1×EV₁, w2×EV₂, w3×EV₃
9. else:
10. Evalue_fin = average of values with the same level
11. return Evalue_fin
End

Table 7
The data set structure

SET OF INSTANCES % (Proportion)

Training 700 70%

Development 200 20%

Testing 100 10%

Risk level 1 133 13.3%

Risk level 2 397 39.7%

Risk level 3 347 34.7%

Risk level 4 104 10.4%

Risk level 5 19 1.9%

SET	OF INSTANCES	% (Proportion)
Training	700	70%
Development	200	20%
Testing	100	10%
Risk level 1	133	13.3%
Risk level 2	397	39.7%
Risk level 3	347	34.7%
Risk level 4	104	10.4%
Risk level 5	19	1.9%

Table 8

The results of precision, recall, and F-measure

Method	TP	FN	FP	Precision	Recall	F-measure
ICS-BPNN	88	9	3	96.7%	90.7%	93.6%
FIS	83	7	10	89.2%	92.2%	90.7%
GA-BPNN	85	5	10	89.5%	94.4%	91.9%
FNN	86	8	6	93.5%	91.5%	92.5%
FRAMB	92	5	3	96.8%	94.8%	95.7%

We found that in some cases, the system risk values given by the three expert teams are not at the same risk level. Therefore, we designed a divergence resolution algorithm to obtain the final system risk value, as shown in algorithm 1. It takes the risk values EV₁, EV₂, and EV₃ given by the expert teams as inputs and calculate the risk levels of these values. If the risk levels are all at the same level, the average value of EV₁, EV₂, and EV₃ is taken as the final risk value. If the risk levels are not at the same level, the algorithm calculates the deviation degrees m1, m2, and m3 and the weights w1, w2, and w3. Then it takes the weighted sum of EV₁, EV₂, and EV₃ as the final system risk value. If only two risk values are at the same risk level, the average value of the two risk values with the same risk level is taken as the final system risk value. Also, we obtain the importance of different risk criterion by analyzing the average risk values of risk factors in different system risk levels, as shown in Fig. 6.

Fig. 6

The average risk values of risk factors in different system risk levels.

6.2 Evaluation

In this section, we carry out comparative experiments to evaluate the effectiveness of FRAMB. All experiments were performed on the computer with the following configuration: the operating system is Windows 10, the CPU is i7-8550, the memory is 8G, and the code environment is Python 3.6.8. We systematically evaluate the effectiveness and accuracy of FRAMB by comparing it with the following five methods.

*ICS-BPNN [3] is an evolutionary algorithm that improved cuckoo search (ICS) to pre-train a back-propagation neural network (BPNN), which can surmount the defect of falling into the local minima to some extent.

*The risk assessment method in paper [11] is based on the Fuzzy Inference Model (FIS). It can produce risk assessment results based on the four risk factors.

*GA-BPNN [21] is an evolutionary algorithm, which uses a genetic algorithm (GA) to optimize the BPNN, and judges the accuracy by comparing the evaluation value with the target value.

*FNN [24] is a risk assessment method based on a fuzzy neural network (FNN), which uses a comprehensive evaluation algorithm and structural adjustment mechanism.

In addition to the above four algorithms, we also compare the expert assessment results with the risk assessment results given by FRAMB to evaluate whether man-machine cooperation can help experts improve the accuracy of risk assessment results.

In this section, we first carry out accuracy and stability experiments. The methods ICS-BPNN, GA-BPNN, FNN, and CM-NN are trained independently ten times, and the models with the highest accuracy are selected. For example, we train NN_Idx and NN_Rul in the CM-NN model and select the models shown in Fig. 7 and Fig. 8. The training accuracy of the two models is about 90%, and the testing accuracy is about 85%. We use 100 pieces of data in the test set for testing, and define the results within the range of the actual risk value ±0.3 that are accurate. We compare the accuracy of five methods, and the results are shown in Fig. 9 assessmentresults. To evaluate the stability of FRAMB, we conduct five experiments, and 20 pieces of data samples are randomly selected from the test set in each time. The above five methods are still used for comparison, and the experimental results are shown in Fig. 10.

Fig. 7

Training process of NN_Idx

Fig. 8

Training process of NN_Rul

Fig. 9

Accuracy comparison of five methods

Fig. 10

Results of multiple experiments

As shown in Fig. 9 and Fig. 10, FRAMB has good accuracy and stability. For each case, the accuracy of FRAMB is slightly higher than that of other methods. It is shown that the FRAMB method is feasible to improve the accuracy of risk assessment by using man-machine cooperation and fitting the upper and lower bounds. At the same time, FRAMB has flexibility and scalability. For example, enterprises can increase the number of experts according to actual needs.

Table 9 shows the precision, recall, and F-measure value of the assessment results. The precision, recall, and F-measure of FRAMB are higher than other methods, and the precision and recall are relatively close. This is because FRAMB uses the method of fitting the upper and lower bounds to reduce the subjective evaluation deviation of experts. Combined with the actual meaning, we find that the number of false negatives and the number of false positives of FRAMB are small and close, which means FRAMB has good assessment centrality.

6.3 Analysis of upper and lower bounds fitting

In this section, we verify the feasibility of using the assessment values given by DFAHP and CM-NN as the upper and lower bounds of the system risk value. We use 100 pieces of data in the test set and take the data tag value increased by 0.4 as the upper bound value, and the data tag value reduced by 0.4 as the lower bound value. At the same time, we define the upper bound value ± 0.2 and the lower bound value ± 0.2 as the upper bound interval and the lower bound interval. If the assessment values given by DFAHP and CM-NN are located in the upper and lower bounds intervals, the two models fit the upper and lower bounds well. The analysis results of the assessment values of DFAHP and CM-NN are shown in Fig. 11.

As shown in Fig. 11, all of the DFAHP assessment values are greater than the actual risk values, and all of the CM-NN assessment values are less than the actual risk values. Also, 79% of the assessment values of DFAHP fall within the upper bound interval, and 82% of the assessment values of CM-NN fall within the lower bound interval. This shows that DFAHP and CM-NN have a good fitting to the upper and lower bounds of system risk assessment value.

Fig. 11

The analysis results of the assessment values of DFAHP and CM-NN.

To illustrate the rationality of the reference interval, we selected three example systems: system 1 with risk assessment level of 1, system 2 with risk assessment level of 2, and system 3 with risk assessment level of 3. We trained ICS-BPNN [3], GA-BPNN [21] and FNN [24] for 5 times, and obtained a total of 15 assessment models. At the same time, FIS [11] and the assessment values of experts are also considered. We use 0.5 as a step to divide the risk interval from 0 to 5 and count the number of assessment values in each step interval. Further, the histogram and normal distribution of the evaluation values of the three systems are drawn according to the statistical results, as shown in Fig. 12.

Fig. 12

Histogram and normal distribution of systems.

After statistical analysis, we express the normal distributions as N(0.64, 0.07), N(1.52, 0.1), and N(2.56, 0.07). Further, we can obtain 80% confidence interval: (0.30, 0.97), (1.11, 1.92), and (2.22, 2.91). Also, 95% confidence interval can be obtained: (0.11,1.16), (0.88,2.14), and (2.02,3.10). in these cases, the reference intervals given by CM-NN and DFAHP are (0.21,1.15), (1.02,2.10), and (2.12,3.08), which are between 80% and 95% confidence intervals. Therefore, it is feasible to take the assessment values given by CM-NN and DFAHP as the lower and upper bounds of the system assessment value.

7 Conclusion

In this paper, we propose FRAMB, a man-machine cooperation risk assessment method based on fitting upper and lower bounds. FRAMB follows the ISO/IEC 27005 standard, which can give more accurate and objective risk assessment results in the periodic assessment of small and medium-sized enterprises. We construct a risk assessment criterion including four categories and sixteen risk factors, which covers all kinds of risk factors in the information system. On this basis, by using the risk characteristics calculation method, we express the risk of the system as a 16-dimensional characteristic. Then the lower and upper bounds of the system assessment value can be calculated by CM-NN and DFAHP. Also, experts can refer to this interval to check whether there are human errors in the assessment process. The experiments show that FRAMB has high precision (96.8%) and recall (94.8%) and the example reference intervals are between 80% and 95% confidence intervals. Hence, we can consider that FRAMB can give accurate and objective results of system risk assessment by integrating the expert assessment value, the upper bound value, and the lower bound value. In future research, we will further improve the automation of FRAMB by using an expert system to give the characteristics of system risk, and reduce the influence of experts’ subjective factors and dependence on experts’ experience.

Footnotes

Acknowledgments

This work was supported by the National Natural Science Foundation of China (General Program) under Grant No.61572253.

References

CVE list. http://cve.mitre.org/data/downloads/index.html/ ,2022.

ISO/IEC 27005. Information technology —Security techniques —Information security risk management. Technical Report 2018 edition, ISO/IEC, Vernier, Geneva, Switzerland, 2018. https://www.iso.org/standard/54534.html.

, Bi

, Chen

, Miao

, Liu

and Tang

, An improved information security risk assessments method for Cyber-Physical-Social computing and networking, IEEE Access 6 (2018), 10311–10319.

Shamala

, Ahmad

, Zolait

A.H.

and Sahib

S.b.

, Collective information structure model for information security risk assessment (ISRA), Journal of Systems and Information Technology 17 (2015), 193–219.

Fattahi

, Tavakkoli-Moghaddam

, Khalilzadeh

, Shahsavari-Pour

and Soltani

, A novel FMEA model based on fuzzy multiple-criteria decision-making methods for risk assessment, Journal of Enterprise Information Management 33 (2020), 881–904.

G.M.

, Chen

Z.H.

, He

X.Z.

, Li

J.P.

A method of security evaluation based on fuzzy mathematics. In 2008 International Conference on Apperceiving Computing and Intelligence Analysis volume pp (2008), pp. 106–109.

Liou

T.S.

, Jiun

and Wang

, Ranking fuzzy numbers with integral value, Fuzzy Sets and Systems 50 (1992), 247–255.

Wen

T.C.

, Chung

H.Y.

, Chang

K.H.

and Li

Z.S.

, A flexible risk assessment approach integrating subjective and objective weights under uncertainty, Engineering Applications of Artificial Intelligence 103 (2021), 104310.

Wang

, Chen

, Feng

, Di

, Liu

, Zhao

and Sui

, Research on network security situation assessment and quantification method based on analytic hierarchy process, Wireless Personal Communications 102 (2018), 1401–1420.

10.

Zhang

, Zhao

, Yang

, Gao

and Xie

, Research on privacy security risk assessment method of mobile commerce based on information entropy and Markov, Wireless Communications and Mobile Computing 2020 (2020), 8888296:1–8888296:11.

11.

Alali

, Almogren

A.S.

, Hassan

M.M.

, Rassan

I.A.

and Zakirul Alam Bhuiyan

Md.

, Improving risk assessment model of cyber security using fuzzy logic inference system, Computer Security 74 (2018), 323–339.

12.

Zhao

S.L.

, Hao

R.X.

and Peng

S.L.

, Reliability Assessment of Some Regular Networks, The Computer Journal 64 (2019), 1–6.

13.

Xue

, Yuan

, Jiang

and Cai

B.Y.

, Difference measure method of risk probability distribution based on moment generating function and fuzzy data stream clustering, Journal of Information Science and Engineering 35 (2019), 1397–1417.

14.

Coffey

and Claudio

, In defense of group fuzzy AHP: A comparison of group fuzzy AHP and group AHP with confidence intervals, Expert Systems with Applications 178 (2021), 114970.

15.

Aguayo

E.A.

, Mateos

and Jimenez

, A new dominance intensity method to deal with ordinal information about a DM’s preferences within MAVT, Knowledge Based Systems 69 (2014), 159–169.

16.

Verma

and Rajasankar

, A thermodynamical approach towards group multi-criteria decision making (GMCDM) and its application to human resource selection, Applied Soft Computing 52 (2017), 323–332.

17.

Wang

Z.C.

, Ran

, Chen

, Yang

and Zhang

, Group risk assessment in failure mode and effects analysis using a hybrid probabilistic hesitant fuzzy linguistic MCDM method, Expert Systems with Applications 188 (2022), 116013.

18.

Esangbedo

M.O.

, Bai

, Mirjalili

S.M.

and Wang

, Evaluation of human resource information systems using grey ordinal pairwise comparison MCDM methods, Expert Systems with Applications 182 (2021), 115151.

19.

Yener

and Can

G.F.

, A FMEA based novel intuitionistic fuzzy approach proposal: Intuitionistic fuzzy advance MCDM and mathematical modeling integration, Expert Systems with Applications 183 (2021), 115413.

20.

Martinez-Garcia

, Zhang

, Suzuki

and Zhang

Y.D.

, Deep recurrent entropy adaptive model for system reliability monitoring, IEEE Transactions on Industrial Informatics 17 (2021), 839–848.

21.

Wang

and Bi

, Risk assessment of knowledge fusion in an innovation ecosystem based on a GA-BP neural network, Cognitive Systems Research 66 (2021), 201–210.

22.

Wang

, Zhu

, Shen

, Lin

, Liu

and Zhao

, A network security risk assessment method based on a B_NAG model, Computer Systems: Science Engineering 38 (2021), 103–117.

23.

, Guo

, Kuang

, Meng

, Hu

and Shi

, I-HMM-Based multidimensional network security risk assessment, IEEE Access 8 (2020), 1431–1442.

24.

Kumar

, Wang

and Song

, E-Commerce credit risk assessment based on fuzzy neural network, Computational Intelligence and Neuroscience 2022 (2022), 3088915, 1.

25.

Mills

, Hu

and Min

, Multi-task federated learning for personalised deep neural networks in edge computing, IEEE Transactions on Parallel and Distributed Systems 33 (2022), 630–641.

26.

ISO/IEC 27001. Information technology —Security techniques —Information security management systems —Requirements, Technical Report 2013 edition, ISO/IEC, Vernier, Geneva, Switzerland, 2013. https://www.iso.org/standard/54534.html.

27.

Goodfellow

, Bengio

, Courville

Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.

28.

Gao

, Guo

, Yu

, Li

RBF-SVM and its application on network security risk evaluation, 2008 4th International Conference on Wireless Communications, Networking and Mobile Computing, (2008), pp. 1–4.

Process	Expert assessment method	NN assessment method	FRAMB method
a) Risk	i. Build an expert team.	i. Determine assessment	i. Determine assessment criterion.
assessment	ii. Develop an assessment	criterion.	ii. Use data set to train and get the
preparation	plan.	ii. Use data set to train	risk lower bound assessment model.
	iii. Determine assessment	and get the assessment	iii. Construct risk upper bound
	criterion and method.	model.	assessment model.
b) Risk	i. Asset analysis.	i. Determine risk items	i. Analyze overall deployment
factor	ii. Threat analysis.	according to assessment	and functions of system.
assessment		criterion.	ii. Determine risk items according
c) Risk	i. Vulnerability analysis	ii. Assess the value of	to assessment criterion, and
determination	ii. Derive risk severity.	risk items.	analyze assets, threats and
		iii. Obtain system	vulnerabilities.
		risk characteristics.	iii. Assess the value of risk items.
			iv. Obtain system risk characteristics.
d) Risk	i. Compare with risk	i. Input risk characteristics	i. Input risk characteristics into
assessment	assessment criterion.	into the assessment model.	the assessment models.
	ii. Conduct risk assessment.		ii. Get the upper and lower bounds
	iii. Determine risk value	ii. Obtain risk value	of system risk assessment value and
	/ risk level.	/ risk level.	the corresponding risk reference
			interval.
			iii. Get the expert assessment result.
			iv. Obtain the weights of upper value,
			lower value and expert value.
			v. Get the final risk value / risk level.
e) Risk	i. Determine whether to accept the risk.
control	ii. Give risk control methods and suggestions.

FRAMB: A man-machine cooperation risk assessment method for intelligent network information systems

Abstract

Keywords

1 Introduction

3 Design of the FRAMB risk assessment method

3.1 Risk assessment framework of the FRAMB

4.1 Risk Characteristics Calculation Method

6.1 Dataset construction

Table 7 The data set structure SET OF INSTANCES % (Proportion) Training 700 70% Development 200 20% Testing 100 10% Risk level 1 133 13.3% Risk level 2 397 39.7% Risk level 3 347 34.7% Risk level 4 104 10.4% Risk level 5 19 1.9%

Footnotes

Acknowledgments

References

Table 7
The data set structure

SET OF INSTANCES % (Proportion)

Training 700 70%

Development 200 20%

Testing 100 10%

Risk level 1 133 13.3%

Risk level 2 397 39.7%

Risk level 3 347 34.7%

Risk level 4 104 10.4%

Risk level 5 19 1.9%