A quality evaluation method for intelligent traffic management system based on business data flow

Abstract

Intelligent Traffic Management System (ITMS) is a complex and intelligent cyber-physical system (CPS) with multi-subsystem interaction, which plays a significant role in traffic safety. However, the quality evaluation requirements of ITMS, particularly its running quality, cannot be satisfied by the current quality evaluation metrics. Moreover, the present ITMS evaluation techniques are arbitrary. The effectiveness of road traffic is impacted because ITMS quality cannot be adequately assured. To fill this gap, this paper proposes a quality evaluation (QE) methodology based on the ITMS business data flow. First, the ITMS QE dimension extraction process was introduced to describe the ITMS architecture and activities; then the new evaluation indexes including intelligence, complexity and interactivity were proposed and an ITMS QE model was established; further through the measurement of metrics elements, the quality score of the indicators were calculated; finally a prototype tool was developed to verify the efficacy and practicability of the method. The results showed that the proposed method has the advantages of accurate problem tracking and decrease decision-making uncertainty. This is applicable to the ITMS QE in various operational scenarios.

Keywords

Intelligent traffic management system complex system multi-system interaction quality evaluation

1 Introduction

Intelligent traffic management system (ITMS) was established for the traffic management department used for traffic control, traffic management and traffic decision-making. It achieved efficient, comprehensive and scientific management of the road traffic system using technologies and methods of intelligent transportation system (ITS) [1]. On the one hand, ITMS is a complex cyber-physical system (CPS) composed of a series of independent and interrelated subsystems, which including dozens of intelligent sensors, heterogeneous network communication subsystems and multiple central application subsystems [2]. On the other hand, software running in ITMS cause issues such as high data dependencies, uncertain results, network interruptions, system outages, etc. It could lead the failure of the whole system, which would have an impact on the road safety, public traffic management, as well as the internal business of the traffic control department. Therefore, improving the quality of ITMS is one of the main tasks that the traffic control department constantly focuses on.

The software or system product quality model covers both abstraction and generalization of the quality attributes, which may serve as a reference for evaluation. Evaluation of product quality requires the establishment of a consistent baseline. At present, the International Organization for Standardization (ISO) standards [3 –5] are used for ITMS QE. But in the evaluation process, there are the following deficiencies. Firstly, the general quality models do not sufficiently handle the unique quality characteristics of intelligent transportation. It lacks proper testing and verification procedures. Analyzing and tracing problems in ITMS is difficult. Secondly, the assignment method about the measure elements’ weight is highly subjective. Thirdly, the traditional product-based evaluation study is based on a single software, which ignores the integration and interaction of the entire system. The evaluated results cannot adequately reflect the quality of the ITMS.

In this paper, a novel evaluation model and quantitative method based on ITMS business data flow is introduced. First, a data processing flow diagram is created to depict the relationship between the ITMS architecture and the business. Then, based on the ISO/IEC 25010, the corresponding metric elements for the intelligence, complexity and interactivity are proposed. The ITMS QE model is constructed. Finally, a series of quantitative indicator scoring methods to evaluate the quality of ITMS are provided. The verification of the applicability and effectiveness of proposed method has been down by evaluating the quality of ITMS of different scales, comparing the quality and actual operation effect of different system evolution versions, and comparing with similar methods.

The rest of this paper is organized as follows: Section 2 reviews some related works to quality evaluation and explains the shortcomings of the existing methods. Research questions are set up; Section 3 introduces ITMS architecture and QE requirements; Section 4 discusses the six steps of proposed method in details; Section 5 finds the answers of research questions in Section 2 by case studies and results analysis and Section 6 illustrates the advantages of the proposed method and concludes this work.

2 Literature review

In this section, some related works to quality evaluation for individual software and quality evaluation method for complex systems are reviewed to find the shortcomings of the existing methods, so as to clarify the purpose and direction of this research.

2.1 Quality evaluation for individual software

The existing quality evaluation of a single software relies on the software quality model. For the general software, the traditional Boehm model [6], McCall model [7] and subsequent models [8 –11] are often used in quality evaluation. For the intelligent software, they have the characteristics such as random input and output, predicting difficulty of all scenarios and continuous learning process from past behavior. Scholars have been studying on to find an effective QE method. Wang et al. [12] proposed a method of research on the quality analysis and prediction method of smart energy meter software. They used the data as the sample data for modeling and used the XGBoost algorithm to establish the quality analysis model. Li et al. [13] pointed out the difficulties of intelligent software quality evaluation from four aspects, and exemplified how to design intelligent feature evaluation of autonomous vehicles. ITMS contains lots of intelligent software. However, because of the interaction and collaboration of the whole system, the quality evaluation of an individual software can not reflect the quality level of the whole system, so it is necessary to propose an evaluation method for the entire system.

2.2 Quality evaluation for complex systems

A complex system refers to a large-scale software system formed by the continuous integration and mutual coupling of a large number of subsystems. Complex system is closely related to the social and physical systems. The behavior of the whole system is difficult to characterize by the simple superposition of the characteristics of subsystems [14]. For the complex software systems quality assessment, scholars have also conducted some researches. For example, in the literature [15], the method of product quality risk assessment in complex manufacturing system has been studied. The Fuzzy-Petri net model of risk assessment is established based on selected evaluation index of complex manufacturing system. Liu et al. [16] proposed a standard system of three-dimensional interface structures with standard types, system levels, and professional classification for electromechanical systems. It included 3 sub-systems and 5 levels. In the field of traffic management, Feng et al. [17] constructed the evaluation index system of the highway ITMS using the fuzzy comprehensive evaluation method to evaluate its construction and application level. It focused on the evaluation of the social benefits brought by ITMS. In order to ensure the running quality of ITMS, it is necessary to evaluate its operation characteristics, system quality and business processing objectively, so as to improve the level of road safety management.

2.3 Quality evaluation methods

Quantitative quality evaluation methods can be divided into metric-based evaluation and model-based evaluation. For metric-based method, Lu et al. [18] proposed QE metrics for safety-critical software testing, which include compliance, stability, adequacy, and effectiveness. AHP was used to assign weights of each index. These weights are fixed. The results obtained may not be accurate if the running scenario changes. Zhang et al. [19] developed a data mapping between the software quality evaluation system and the software testing process. They developed the model for quality analysis. For model-based method, the model is the core of the evaluation activity. The existing software evaluation models mainly include hierarchical decomposition model, the threat tree model, intelligent evaluation model and mathematical evaluation model. The mathematical evaluation model uses fuzzy theory to quantify qualitative indicators, and solves the problem of a single indicator and original information for the existing evaluation model. Sun et al. [20] used the fuzzy models to evaluate the software testing quality. This method has shortcomings. When the index set is large and the sum of weight vector is 1, the weight coefficient of relative membership degree is often small, the weight vector does not match the fuzzy matrix. The result can be a failure to distinguish who has higher membership and even a failure to evaluate. Awasthi and Omrani [21] combined the AHP with evidence theory and proposed a DSAHP multi-attribute decision-making model. This model can not only reduce the number of pairwise judgments and consistency checks, but also provide a good solution when the information is insufficient.

In order to cover the shortage of ITMS quality evaluation and improve the existing evaluation methods, we propose an evaluation method based on ITMS business data flow. Three research questions are put forward as follows to clarify the research direction and targets.

RQ1: Whether the new quality model can cover all attributes of ITMS?

RQ2: Can ITMS QE method provide interpretable results in different operational scenarios?

RQ3: What are the advantages of the proposed method compared with other similar methods?

The purpose of RQ1 is to explore the applicability of the ITMS QE model; the purpose of RQ2 is to explore the effectiveness of ITMS QE in different operational scenarios; the purpose of RQ3 is to explore the performance of ITMS QE in practice.

3 ITMS quality evaluation requirements

In this section, the architecture and software types deployed in ITMS is introduced and the quality evaluation requirements of ITMS are summarized.

3.1 ITMS architecture

With the development of industrial informatization, as a typical CPS, the architecture of ITMS is becoming more and more complex, characterized by large-scale, integration, refinement and coupling. The architecture is composed of sub-modules including the physical layer, cyber layer and application layer [22]. Several communication and sensing devices are mostly employed in the physical layer, with embedded software serving as the primary software type. Fiber optic or wireless networks are used in the cyber layer to support system communication and provide security protection. Middleware is the most common sort of software. Functions including command and dispatch, service administration and intelligence analysis are all developed in the application layer. Desktop software, mobile software, and embedded software are the three main forms of software. New software forms, such as big data software, cloud computing software, artificial intelligence (AI) software, etc., are gradually incorporated into each layer with the deployment of sophisticated technologies like the Internet of Things (IoT), big data, and cloud computing.

3.2 ITMS quality requirements analysis

High levels of safety, reliability and availability are required by ITMS. Travel safety and precise traffic scheduling are indicators of system quality. The architecture of the system is becoming increasingly complex due to the advancement of ITS development and the ongoing expansion of information system scales. At the same time, there are more mistakes or flaws in the development process, making it harder to ensure the quality of the system. This section describes the quality evaluation requirements of ITMS as an intelligent, complex and interactive system.

3.2.1 Quality requirements of intelligent systems

ITMS is an intelligent system that can recognize and analyze the key traffic management objectives automatically. It can detect illegal behaviors based on traffic participants such as "people, vehicles and roads". Its existence improves traffic efficiency and optimizes traffic flow. However, traditional quality models lack evaluation indicators for intelligence, resulting in ITMS QE blind spots. Quality issues such as inaccurate identification and judgment, have resulted in low traffic enforcement efficiency, inability to alleviate traffic congestion in a timely manner and low satisfaction with public services. The intelligence of ITMS are primarily reflected in traffic management and control decision-making based on multimedia input, as well as the realization of image recognition of vehicles, faces, license plates. Since AI applications are uncertain and probabilistic, the quality evaluation of intelligent software differs from traditional software.

3.2.2 Quality requirements of complex systems

ITMS is a typical complex system. It possesses not only the typical characteristics of complex systems, such as complexity, uncertainty, and diversity, but also real-time, growth, rapidity, interactivity, and other important features. The presence of a single point of quality clearance does not imply that the entire system is free of quality risks. Furthermore, users are concerned not only with whether the system can run normally, but also with how well the system functions. The majority of existing quality evaluation methods are designed for a single piece of software or a single product. The functional performance technology of many pieces of equipment and subsystems is described in national or industry standards. For instance, traffic lights, traffic signals, traffic flow information collection systems, road vehicle intelligent monitoring and recording systems and automatic red light-running recording systems. However, the development of quality standards for the whole system is still blank.

3.2.3 Quality requirements formed by the interaction and integration of multiple subsystems

Complexity, high real-time demands, huge data transmission and processing, data security are all features of the ITMS’s operational process. The exchange of data and information between subsystems is substantial. Nevertheless, because each subsystem uses a separate set of protocols and there are issues with information security, it is not possible to fully communicate and share vital data. The functional logic and architectural design of the ITMS are getting more and more sophisticated as it moves toward integration and intelligence. There are more and more connections between subsystems and diverse physical and social entities. In order to improve the ITMS quality and standardize the corresponding access thresholds or limits, action must be taken immediately.

4 Methodology

In this section, the business data flow based ITMS QE method is elaborated. The business data is the core of road traffic management. ITMS consists of thousands of front-end devices, network nodes and back-end systems. Its running effectively and realization of traffic management functions, which cannot be separated from the interaction of data. According to the architecture of ITMS, the logical framework composed of IOT sensing equipment, support system and application system reflects the data flow process. That is, the whole closed loop of data acquisition, transmission and application is the operation mode of ITMS and the common feature of each subsystem. In this paper, business data flow is used as the basis of the common index of ITMS quality evaluation to reduce the workload of evaluating the quality of a large number of devices and subsystems. This method focuses on the evaluation of the running quality of the whole ITMS, so as to improve the efficiency of traffic management and ensure road traffic safety. The ITMS QE method consists of following stages: determining evaluation dimensions, building attribute knowledge base, indentifying evaluation indexes, defining evaluation metrics, assigning weights and computing quality evaluation results.

4.1 Determining evaluation dimensions

The key of ITMS QE is to abstract the operation mode and key factors of the system based on the application effect to form a common evaluation model. The dimensions’ extraction process is shown in Fig. 1. There dimensions are selected, including data acquisition, data transmission, data application as the basis of the index of quality evaluation. The evaluation dimensions based on business data flow can weaken the function and performance indicators of a single device, while the evaluation method based on data quality can reflect the overall intelligence perception ability, information interaction ability and application benefits of ITMS.

Fig. 1

ITMS quality evaluation dimensions extraction process.

4.1.1 Data acquisition

The ITMS puts forward the requirement of the whole space-time perception of traffic elements. The accurate road traffic monitoring data (e.g. traffic flow at a particular point of the road network, average travel time for a particular road section, number of passengers boarding a transit line, etc.) and stable front-end data acquisition equipment (e.g. cameras used in security video surveillance, motor vehicle speed detector, traffic flow detector, etc.) are the key elements of this evaluation dimension. The evaluation of traffic data perception and acquisition technology can reflect the intelligent characteristics of ITMS.

4.1.2 Data transmission

Traffic data is complicated and diversified, which can be divided into dynamic and static data, structured and unstructured data. ITMS plays an important role in the fusion and extended analysis of data from different sources. Data storage capacity, computing capacity and AI algorithm are the most capabilities of ITMS. Therefore, in the dimension of data transmission, it is necessary to evaluate the key elements of data transmission, convergence, storage, fusion and calculation to assess the complex characteristics of ITMS.

4.1.3 Data application

The essential criterion for the QE of ITMS in terms of solving road traffic problems is to assist in the realization of the traffic management function and to ensure the effect of safety management. The evaluation of data application dimensions can verify that the system has complete functionalities based on multi-source heterogeneous information analysis, diagnosis, planning, decision making and the production of management plans. Simultaneously, ITMS could be validated to have an interactive mode based on data and information, as well as the ability to provide users with accurate and active services. As a result, it meets the requirement for ITMS multi-system interactive quality evaluation.

4.2 Building attribute knowledge base

To construct the ITMS attribute knowledge base (AKB), we summarized the qualities that affect the quality of ITMS at all levels based on the ITMS QE dimensions and the ITMS components. The database construction provides source support for identifying evaluation indexes. There are two types of ITMS attributes: general attributes and proprietary attributes. General attributes indicate the properties of the software ontology. Functionality, dependability, security and maintainability are examples. Proprietary attributes are associated with the software’s application business. The ITMS AKB is created through extensive study and expert polls. The ITMS AKB’s construction logic is depicted in Fig. 2.

Fig. 2

The construction logic of the ITMS attribute knowledge base.

The“3-3-4”construction method is adopted, that is, 3 object categories(sensing device, support service and application system), 3 nodes (data acquisition, data transmission and data application), 4 business (road surveillance, traffic control, illegal disposal, information services). The pyramid ITMS AKB was established.

The ITMS AKB is divided into three sections: the general attribute library, the proprietary attribute library and the mapping section. The general attribute library is divided into two parts: the basic indicator set and the assessment method set. The proprietary attribute library is a collection of software quality attributes developed specifically for specific application systems. It is necessary to extract application knowledge and user requirements according to ITMS software types. It offers user-defined indicators and assessment methods to add particular criteria and can be mapped with the general attribute library. When the general attribute library is applied to the quality measurement of a specific application system, the mapping section relates to the selection, deletion, expansion, or change of the index set and the calculation method.

4.3 Indentifying evaluation indexes

The evaluation attributes are further selected and decomposed based on the evaluation dimensions and AKB to determine the ITMS QE indexes. The QE indexes are derived from the AKB, which specifies the evaluation first level index of functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability and portability based on ISO/IEC 25010. ITMS quality requirements can be met with the new attributes including intelligence, complexity and interactivity. Figure 3 depicts the corresponding relationship between ITMS evaluation dimension and quality indexes. The second level indexes are refinements of the first level indexes that can better reflect ITMS characteristics. The extraction of the second level indexes does not focus on a certain type of software and system, but establishes the corresponding relationship from the business application and data flow.

Fig. 3

ITMS quality evaluation index system.

Furthermore, as new devices, technologies, and functions are added to ITMS, we can add new ITMS characteristics to the AKB. As a result, the ITMS quality index system allows for user customization to meet the actual evaluation requirements of various types of software as well as business data flows. Figure 4 shows the dynamic process of index system improvement.

Fig. 4

The dynamic process of index system improvement.

According to the software quality evaluation requirements, the quality attributes are extracted from the software quality knowledge base. If the attributes in the library are suitable for the needs of evaluation, then layer-by-layer matching is performed, and metrics are selected or added. If the attributes is absent, then new metrics should be added through layer-by-layer analysis. Finally, an index tree is generated. All added attributes and indicators could be reclassified into the software feature library to form a closed loop. ITMS quality index system could adjust quality attributes and metrics freely, which provides a flexible and feasible method to evaluate the complexity requirements of the system.

4.4 Defining evaluation metrics

ISO/IEC 25023 provides metrics for the quality attributes. It assists the quantitative evaluation of system quality. In this way, we defined metrics for intelligence, complexity and interactivity applying ITMS QE, including: TF, TP, DC, ER, TL, AUR, OW, SC, MC, VF, AIC, DTE, HCI, DCT, DFC, DR, DI. The metric details are shown in Table 1. The measurement of intelligent characteristics mainly evaluates the intelligent automation ability of ITMS such as self-management and self-specification, the accuracy of the adopted algorithm and the advanced degree of the model. The measurement of complex characteristics mainly evaluates the operability of the external functions of the ITMS and the analyzability of the internal architecture. The measurement of subsystem interaction characteristics mainly evaluates the stability of ITMS data transmission, human-computer interaction ability and effectiveness of interaction.

Table 1
ITMS quality evaluation metrics

First level index Second level index Metric Calculation method Value range

Intelligence Autonomy Takeover frequency (TF) X=A/B A=Number of manual takeover B=Task execution time [0,+∞)

Target perception (TP) X=A/B A=Number of targets that can be adapted B=Total number of targets [0,1]

Decision correctness (DC) X=A/B A=Number of input behavior data B=Number of correct decision behavior [1,+∞)

Accuracy Element recognition (ER) X=A A=Number of recognized elements correctly [0,+∞)

Timeliness (TL) X=A/B A=Number of processed data on time B= Total Number of data need to be processed [0,+∞)

Advancement AI algorithm using rate (AUR) X=A/B A=Number of AI algorithm B=Total number of functions [0,1]

Complexity Maneuverability Operator workload (OW) X=A/B A=Manual takeover time B=Task execution time [0,1]

Analyzability Standards compliance (SC) X=A/B A=Number of features that meet the standard requirements B=Total number of features required by the standard [0,1]

Module coupling (MC) X=A/B A=Number of modules coupled B=Total number of modules [0,1]

Visible functions (VF) X=A A=Number of visible functions [0,+∞)

Interactivity Stability Anti-interference capability (AIC) X=A/B A=Number of interference types that can be adapted

B=Number of interference types that need to be adapted [0,1]

Data transmission efficiency (DTE) X=A A=Speed of data transfer [0,+∞)

Visuality Human-computer interaction (HCI) X=A A=Number of human-computer interaction interfaces [0,+∞)

Interoperability Data collection time (DCT) X=A A=Data collection time [0,+∞)

Data format compliance (DFC) X=A/B A=Number of data items that meet the required format B=Number of data items [0,1]

Data reused rate (DR) X=A/B A=Number of reused data B=Number of data [0,1]

Data inheritance (DI) X=A A=Number of reserved interfaces [0,+∞)

First level index	Second level index	Metric	Calculation method	Value range
Intelligence	Autonomy	Takeover frequency (TF)	X=A/B A=Number of manual takeover B=Task execution time	[0,+∞)
		Target perception (TP)	X=A/B A=Number of targets that can be adapted B=Total number of targets	[0,1]
		Decision correctness (DC)	X=A/B A=Number of input behavior data B=Number of correct decision behavior	[1,+∞)
	Accuracy	Element recognition (ER)	X=A A=Number of recognized elements correctly	[0,+∞)
		Timeliness (TL)	X=A/B A=Number of processed data on time B= Total Number of data need to be processed	[0,+∞)
	Advancement	AI algorithm using rate (AUR)	X=A/B A=Number of AI algorithm B=Total number of functions	[0,1]
Complexity	Maneuverability	Operator workload (OW)	X=A/B A=Manual takeover time B=Task execution time	[0,1]
	Analyzability	Standards compliance (SC)	X=A/B A=Number of features that meet the standard requirements B=Total number of features required by the standard	[0,1]
		Module coupling (MC)	X=A/B A=Number of modules coupled B=Total number of modules	[0,1]
		Visible functions (VF)	X=A A=Number of visible functions	[0,+∞)
Interactivity	Stability	Anti-interference capability (AIC)	X=A/B A=Number of interference types that can be adapted
			B=Number of interference types that need to be adapted	[0,1]
		Data transmission efficiency (DTE)	X=A A=Speed of data transfer	[0,+∞)
	Visuality	Human-computer interaction (HCI)	X=A A=Number of human-computer interaction interfaces	[0,+∞)
	Interoperability	Data collection time (DCT)	X=A A=Data collection time	[0,+∞)
		Data format compliance (DFC)	X=A/B A=Number of data items that meet the required format B=Number of data items	[0,1]
		Data reused rate (DR)	X=A/B A=Number of reused data B=Number of data	[0,1]
		Data inheritance (DI)	X=A A=Number of reserved interfaces	[0,+∞)

All the metrics measurement data could get from software artifacts which includes review, testing and actual operation. Review refers to an independent inspection of the conformity of requirement specifications, baselines, standards, processes, instructions, code, contracts and special requirements. Testing includes demonstration verification, experimental verification, software static test and dynamic test. They can be used to calculate most of measurement index values. In actual operation, some measurement data can be obtained, such as resource usage, equipment status and logging.

4.5 Assigning weights

Different roles have very different concerns about the quality of ITMS in different environments and business conditions. Users and designers may be more concerned with functional performance in road traffic monitoring tasks, whereas testers may be more concerned with reliability but have fewer autonomy requirements. To provide more accurate traffic information, the system must be more reliable, adaptable, timely and autonomous for vehicle inspection, traffic control, and traffic judgment tasks. As a result, the weight distribution of indicators should be combined with specific environments and tasks, as well as taking into account the subjective preferences of different roles. The evaluation method in this paper is based on the analytical algorithm. The AHP technique developed in 1980 to solve unstructured problems by Saaty. It is an important Multiple Criteria Decision Making (MCDM) method which is used to determine the most appropriate option among alternative options under multiple criteria and different targets 23]. In practice, we found that the AHP has some flaws when applied. With a large number of evaluation factors, it is easy to make a mistake or fall into a logical contradiction. To avoid the cumbersome process of judging the consistency, this paper proposed a pseudo-optimal consistency matrix to avoid the large amount of calculation and complex process in the consistency verification process.

The principles and models of improved AHP (IAHP) are as follows:

Suppose the real matrix: A = [a_ij], B = [b_ij], C = [c_ij] ∈ R^n×n.

Definition 4.1. If $a_{ij} = \frac{1}{a_{ji}}$ , then A is called the reciprocal matrix; if b_ij = - b_ji, then B is called the antisymmetric matrix.

Definition 4.2. If A is reciprocal matrix and $a_{ij} = \frac{a_{ik}}{a_{jk}}$ , then A is consistent; if B is an antisymmetric matrix and b_ij = b_ik + b_kj, then B is transitive.

Definition 4.3. If there is a transfer matrix C and the value of $\sum_{i = 1}^{n} \sum_{j = 1}^{n} {(c_{ij} - b_{ij})}^{2}$ is minimized, then C is the optimal transfer matrix of B.

Theorem 1. If B is an antisymmetric matrix, then the optimal transfer matrix C of B should be:

$c_{ij} = \frac{1}{n} \sum_{k = 1}^{n} (b_{ik} - b_{jk})$ (1)

Proof. Assume C is transitive, it exists:

x₁, x₂, ⋯ , x_n ⩾ 0, it makes c_ij = x_i - x_j.

Let $\begin{matrix} D & = \sum_{i = 1}^{n} \sum_{k = 1}^{n} {(c_{ik} - b_{ik})}^{2} \\ = \sum_{i = 1}^{n} \sum_{k = 1}^{n} {(x_{i} - x_{k} - b_{ik})}^{2} \end{matrix}$

In order to make C is the optimal transfer matrix of B, D should be minimum, that is: $\frac{\partial D}{\partial x_{i}} = 0$ . $\begin{matrix} \frac{\partial D}{\partial x_{i}} & = 2 {nx}_{i} - 2 \sum_{k = 1}^{n} (x_{k} + b_{ik}) = 0 \end{matrix}$ So: $\begin{matrix} x_{i} & = \frac{1}{n} \sum_{k = 1}^{n} (x_{k} + b_{ik}) \\ x_{j} & = \frac{1}{n} \sum_{k = 1}^{n} (x_{k} + b_{jk}) \\ c_{ij} & = x_{i} - x_{j} = \frac{1}{n} \sum_{k = 1}^{n} (b_{ik} - b_{jk}) \end{matrix}$ □

Theorem 2. If A is a reciprocal matrix,

$B = lg A$ (2) C is the optimal transfer matrix of B, then matrix $A^{*} = 10^{C}$ (3)A^* is a quasi-optimal transfer matrix of A, and it is consistent.

Proof. Since C is the optimal transfer matrix of B, we can know from Definition 3.3: $\sum_{i = 1}^{n} \sum_{j = 1}^{n} {(c_{ij} - b_{ij})}^{2}$ reaches the minimum value.

So:

$\sum_{i = 1}^{n} \sum_{j = 1}^{n} {(lg a_{ij}^{*} - lg a_{ij})}^{2} = \sum_{i = 1}^{n} \sum_{j = 1}^{n} {(c_{ij} - b_{ij})}^{2}$ also reaches the minimum value.

Therefore, A^* can be considered as a quasi-optimal transitive matrix of A. According to the c_ij in Theorem 1: $\begin{matrix} \frac{a_{il}^{*}}{a_{jl}^{*}} & = \frac{10^{c_{il}}}{10^{c_{jl}}} = 10^{\frac{1}{n} \sum_{k = 1}^{n} (b_{ik} - b_{jk})} \end{matrix}$ while,

$a_{ij}^{*} = 10^{c_{ij}} = 10^{\frac{1}{n} \sum_{k = 1}^{n} (b_{ik} - b_{jk})} ∴ a_{ij}^{*} = \frac{a_{ij}^{*}}{a_{ji} i}$

From Definition 3.2, A^* is consistent.□

In the traditional AHP method, the judgment matrix A = (a_ij) _nx_n is a reciprocal matrix, and B = lg A is an antisymmetric matrix. The matrix A^* = 10^{c
_ij} is constructed. From Theorem 2, matrix A^* is a quasi-optimal transfer matrix of A and it is consistent. So, we can calculate the weight value by A^* without the process of consistency check. When solving the eigenvalue (weight) of A^*, we adopt the square-root method. Since the judgment matrix has errors, there is no need to pursue the eigenvector with high precision. The square-root method is actually a hypergeometric average method to find the weight value. The eigenvalues of the quasi-optimal transfer matrix A^* can be made closer to the eigenvalues of the optimal transfer matrix within a certain range. The complete IAHP-based weight assignment algorithm is summarized in Algorithm 1.

Algorithm 1

IAHP-based weight assignment

Input: evaluation index set I = {I₁, . . . , I_m}, expert set S = {S₁, . . . , S_r}

Output: the weight of each evluation index ω

1: Expert S_r compare I_i and I_j to construct a comparison matrix: $D^{(k)} = {[d_{ij}^{(k)}]}_{m \times_{m}}$

2: Use the importance ranking index $r_{i}^{(k)}$ to construct the judgment matrix $A^{(k)} = {[a_{ij}^{(k)}]}_{m \times_{m}}$ , where $\begin{matrix} a_{ij}^{(k)} = {\begin{matrix} r_{i}^{(k)} - r_{j}^{(k)} & r_{i} > r_{j} \\ 1 & r_{i} = r_{j} \\ {[r_{j}^{(k)} - r_{i}^{(k)}]}^{- 1} & r_{i} < r_{j} \end{matrix} \\ r_{i}^{(k)} = \sum_{l = 1}^{m} d_{il}^{(k)} i = 1, \dots, m; k = 1, \dots, r \end{matrix}$

3: Calculate standard deviation of expert evaluation. According to (2): $σ_{ij} = \bar{\frac{1}{S - 1} \sum_{t = 1}^{S} {[b_{ij}^{(l)} - \frac{1}{S} \sum_{i = 1}^{S} b_{i j^{(t)}}]}^{2}} (i, j = 1, \dots, m)$

if σ_ij1 then

5: The evaluation results of the experts are consistent. $b_{ij} = \frac{1}{S} \sum_{l = 1}^{s} b_{ij}^{(l)}$

6: else

7: The evaluation results of experts are widely different. Use optimal transfer matrix method. $b_{ij} = \frac{1}{ms} \sum_{i = 1}^{m} \sum_{l = 1}^{s} {[b_{it}^{(l)} - b_{jl}^{(l)}]}^{2}$

8: According to (1)(3): $A^{*} = 10^{c_{ij}} = 10^{\frac{1}{m} \sum_{l = 1}^{1} (b_{il} - b_{jl})}$

9: Solve the maximum eigenvalue of A^*, the corresponding vector is the weight of each index: $W^{'} = (ω_{1}^{'}, . . ., ω_{m}^{'})$

10: return $W = (ω_{1}, . . ., ω_{m}), W_{i} = \frac{W_{i}^{'}}{\sum_{k = 1}^{m} W_{k}^{'}}$

Using the same method as above, the weight of each evaluation dimensions, quality attributes and indexes is obtained.

4.6 Computing quality evaluation results

After obtain the measured value of selected metric elements. The range of the value is α ∈ [0, + ∞), where +∞ stands for the larger the data of the measurement, the better the results. Using the normalization algorithm [24], the evaluation value could be obtained and the evaluation value is in the range of β ∈ [0, 100]. The specific measured value and evaluation value conversion algorithm are shown in Table 2.

Table 2
The measured value and evaluation value conversion algorithm

Range of measure value Assessment Criteria Evaluation value

[0,1] The closer to 1 the better β = α

[0,1] The closer to 0 the better β = 1 - α

[1,+∞) The closer to 1 the better β = 1/α

0,+∞) The closer to 0 the better β = 1/(1 + α)

0,+∞) The closer to +∞ the better β = 1 -1/(1 + α)

Range of measure value	Assessment Criteria	Evaluation value
[0,1]	The closer to 1 the better	β = α
[0,1]	The closer to 0 the better	β = 1 - α
[1,+∞)	The closer to 1 the better	β = 1/α
0,+∞)	The closer to 0 the better	β = 1/(1 + α)
0,+∞)	The closer to +∞ the better	β = 1 -1/(1 + α)

In order to obtain the evaluation value of each evaluation index, the corresponding evaluation value of the metric element are calculated by weighted average. Based on the evaluation value and weight distribution value of each indicator in the system, the weighted average calculation method is used to determine the quality score of the ITMS through multi-layer compound operations. The quality attribute evaluation results are expressed as: $S_{ijk} = \sum_{k = 1}^{n} Z_{k} ω_{ijk}$ . In the formula, k represents the number of evaluation indexes. According to the obtained evaluation value and weight value, the summation is performed to obtain the score of the quality attributes. Similarly, the evaluation dimension results are expressed as: $S_{ij} = \sum_{k = 1}^{n} Z_{k} ω_{ij}$ . The comprehensive quality score of ITMS system is: $S_{i} = \sum_{k = i}^{n} S_{i} ω_{i}$ . The standard score for the comprehensive evaluation is 100 points. By demarcating the score interval of different result grades, the grade judgment results could be obtained. The quality level of the ITMS can be divided into four levels: A, B, C, and D. Through practical demonstration, the score range of each level is as follows:

Level A - 90 points and above.

Level B - 75 points and above.

Level C - 60 points and above.

Level D - 59 points or less.

5 Application and validation

In order to conduct the ITMS QE efficiently, we have implemented a prototype tool in Java 1 . The interface of the tool is shown in Fig. 5. The tool realization adopts the mode of hierarchical development. The whole system is divided into front-end presentation layer, interface access layer, business service layer and basic data layer. The detailed structure is shown in Fig. 6.

Fig. 5

ITMS quality evaluation prototype tool interface.

Fig. 6

ITMS quality evaluation prototype tool structure.

The front-end presentation layer contains software quality evaluation page and knowledge base management page. It realizes the functions of the ITMS special attribute knowledge base construction, model configuration and automatic assessment. The interface access layer is the bridge between the front-end page and the back-end logic. The business service layer is the realization of the core functions of the system. Including software information data collection service, software quality assessment service, test case management service, query statistics service, etc. It is a technical service for front-end presentation layer. The basic data layer adopts the MyBatis framework [25], including basic database, measurement database, and knowledge database.

In order to answer the questions raised in section 2, we designed 3 case studies.

5.1 Case study #1

We selected ITMS systems from cities of different sizes for quality evaluation. The scale data of ITMS in two cities is shown in Table 3. We represent the two cities with A and B. The data includes the number of devices, services nodes, storage databases, core functions and central management platforms at each stage of data acquisition, transmission and application. We gathered all of the evaluation metric data as well as the evaluation value conversion. The same evaluation index weight distribution is used for City A and City B in order to obtain a comparative evaluation index score.

Table 3
ITMS system scale of two cities\label tab3

Data flow System scale City A City B

Data acquisition Number of devices 18570 3184

Data transmission Number of service nodes 521 95

Number of storage databases 18 6

Data application Number of core functions 42 30

Number of central management platforms 4 2

Data flow	System scale	City A	City B
Data acquisition	Number of devices	18570	3184
Data transmission	Number of service nodes	521	95
	Number of storage databases	18	6
Data application	Number of core functions	42	30
	Number of central management platforms	4	2

5.1.1 Expected results

In general, the larger the city scale, the better the ITMS construction. In terms of intelligence, large-scale ITMS could process more traffic data. The data perception and processing ability of acquisition equipment are good. Its intelligence level is relatively high. In terms of complexity, more hardware and software are integrated in large-scale ITMS. The interaction and association between them make the system more complex. While interactivity is related to the city’s overall traffic operation capacity. To some extent, the traffic congestion reflects the low interactive efficiency of traffic control system. In other words, the higher the complexity of system, the larger the amount of data transferred, and the greater the impact on the interaction ability of the overall system.

5.1.2 Result analysis

The quality score of the ITMS of City A and City B for first level indexes of quality evaluation is shown in Fig. 7. Answer RQ1:the ITMS quality model covers the attributes of ITMS, and all the metric data are measurable.

Fig. 7

ITMS quality evaluation indexes results.

City A’s ITMS is 30.7% more intelligent, 19.8% more complex, and 10.67% less interactive than city B’s. The ITMS quality score of City A is 75.05, which belongs to Level B based on the calculation of the comprehensive quality score. It requires improvement in terms of functional suitability, security, and interactivity. City B has an ITMS quality score of 70.99, putting it in the Level C. This demonstrates that ITMS construction in small cities is not perfect. Answer RQ2: as can be seen, our method is appropriate for evaluating the quality of ITMS on various scales. At the same time, the classification of score levels allow for the identification of quality flaws in various ITMS.

5.2 Case study #2

We carried out a simple system evolution of ITMS in City B. Since City B has a large space for system improvement, it is easier to see the effect of traffic management after system evolution. We select a road in City B to upgrade the related equipment, network system and back-end management platform. The different evolution versions are described as follow:

B-1: Upgrade the control strategy of traffic signal control system from timing control to adaptive control.

B-2: Add a networked traffic video surveillance system.

B-3: Change the Oracle database of traffic data management platform to an HBase.

5.2.1 Results

The QE score impact of the ITMS evolution is shown in Fig. 8. The upgrade of ITMS covers the whole data flow process, including data acquisition, data transmission and data application. Therefore, we use these three evaluation dimensions to conduct a comparative analysis of the system evolution effect.

Fig. 8

The quality evaluation score impact of the ITMS evolution.

We also compared the traffic efficiency of the selected road section. The average vehicle speed before and after the system upgrade at different time periods is shown in Table 4.

Table 4

The average vehicle speed comparison

Periods of time	Average vehicle speed on the road (km/h)
	Before ITMS upgrade	After ITMS upgrade
7:00-8:30	18.23	21.9
17:30-18:30	14.98	16.94
11:00-12:00	26.29	30.67
Average speed	19.83	23.17

5.2.2 Analysis

By comparing the QE results of different evaluation dimensions and measuring the actual road traffic efficiency, the following analysis is obtained to answer RQ1 & RQ2:

Effect of system evolution on ITMS quality. As can be seen from the area of radar map construction, the larger the area, the better the quality. All three system evolution methods can improve the quality of ITMS. At the same time, with the improvement of ITMS quality, the traffic condition of the road has also improved. The average speed increased from 19.83km/h to 23.17km/h.

Verification of evaluation dimension and indexes. For data acquisition evaluation dimension, the B-1 approach has the greatest impact. This is because we upgraded the equipment used in the data acquisition stage directly. Only the change of a single front-end device has little impact on the functional suitability, interactivity and portability of the system. While, the usability and intelligence of the system are improved greatly. The reason is that we have added the intelligent lane detection algorithm in the front-end equipment, which can adjust the phase transformation time of signal control by identifying the traffic efficiency of the lane. While improving the intelligence, the usability has also been improved. The system evolution effect is also consistent with the improvement effect of actual traffic efficiency. For data transmission evaluation dimension, data transmission is associated with the interaction of the whole system. It can be seen that each evolution mode has an effect on the quality improvement in this dimension. Due to the upgrading and addition of equipment, the complexity of the whole system is affected, which also verifies the correctness of our definition of complexity. For data application evaluation dimension, B-1 and B-2 increase the amount of data collected. These data can play a great application value in the decision-making of the system, so the intelligence of the system and the usability of data are improved. B-3 improves the database performance of ITMS. The data security and maintainability of the whole system are also improved.

Optimization of ITMS quality improvement. In a comprehensive comparison, the B-3 method has the greatest impact on the quality evaluation of each dimension, which also indicates that our evaluation method can provide a better solution for the upgrade of ITMS in the case of limited resources.

5.3 Case study #3

This study was designed to make comparison of similar quality evaluation methods. The traditional AHP and fuzzy comprehensive method were selected to make a comparison. We select indexes of data transmission dimension for comparative experiment. Here, we can answer RQ3.

IAHP

Calculating the weight vector of the evaluation indexes in data transmission dimension: \\ ω = (0.341, 0.162, 0.341, 0.078, 0.078). The ITMS quality score of this evaluation dimension is obtained after weighting operation of weights and index scores. The score is 76.9 and the quality level is B.

AHP

Calculating the eigenvector of the largest eigen root of the constructed matrix, that is: \\ I = (6.542, 3.152, 6.542, 1.495, 1.495). After normalization, the quality indexes weight vector H = (0.314, 0.162, 0.341, 0.078, 0.078) is obtained. The maximum characteristic root is: λ = 5.032. When UI = 0, it means the judgment matrix is consistent completely, the larger the UI, the worse the consistency. To measure UI, Saaty [26] introduced the random consistency indicator RUI. By constructing more than 500 pairwise comparison matrices randomly, the values of RUI were obtained as shown in Table 5.

Table 5
Value list of random consistency indicator RUI\label tab5

n 1 2 3 4 5 6 7 8 9

RUI 0 0 0.58 0.9 1.12 1.24 1.32 1.41 1.45

n	1	2	3	4	5	6	7	8	9
RUI	0	0	0.58	0.9	1.12	1.24	1.32	1.41	1.45

When n = 5, according to the consistency ratio formula $UR = \frac{UI}{RUI} = \frac{0.008}{1.12} = 0.007 \leq 0.1$ , the consistency test is satisfied. It can be seen from the above comparative experiments that in determining the weight coefficient of quality characteristics process, the results obtained by using the IAHP method are consistent with those obtained by using the traditional AHP method. Furthermore, the evaluation process of the IAHP is obviously simplified. It improved the efficiency of quality evaluation.

Fuzzy comprehensive method

The quality indexes were evaluated by 20 expert review team members using four evaluation levels of V = {V₁, V₂, V₃, V₄}. Among them, the expert review team consists of 3 software engineering domain experts, 6 traffic management domain experts, 5 software testing engineers, 3 quality managers and 3 ITMS system users. The age distribution of the experts ranged from 28 to 55 years old. With the composition of such experts, who have a wide range of professional fields and are closely related to the design, usage, operation and maintenance of ITMS, making the evaluation results specific and typical. The results are shown in Table 6.

Table 6

The number of votes scored by experts\label tab6

Votes	V ₁	V ₂	V ₃	V ₄
U ₁	3	8	6	3
U ₂	8	6	3	3
U ₃	6	7	5	2
U ₄	2	8	8	2
U ₅	2	7	9	2

The quality index fuzzy evaluation matrix R obtained by the expert evaluation team is: $R = [\begin{matrix} 0.15 & 0.4 & 0.3 & 0.15 \\ 0.4 & 0.3 & 0.15 & 0.15 \\ 0.3 & 0.35 & 0.25 & 0.1 \\ 0.1 & 0.4 & 0.4 & 0.1 \\ 0.1 & 0.35 & 0.45 & 0.1 \end{matrix}]$ According to the formula: Z = W · R, we can calculate the evaluation vetor of indexes.

Performance Efficiency: Z₁ = (0.303, 0.29, 0.407, 0)

Interactivity: Z₂ = (0.286, 0.30, 0.414, 0)

Complexity: Z₃ = (0.253, 0.330, 0.417, 0)

Reliability: Z₄ = (0.272, 0.380, 0.348, 0)

Security: Z₅ = (0.379, 0.540, 0.081, 0))

Calculating the QE dimension vector: Y= W · R = (0.294, 0.357, 0.349, 0), that is, the comprehensive quality belongs to the grade membership degree A : 0294, B : 0.357, C : 0.349, D : 0. According to the principle of maximum membership, the ITMS QE result is grade B, which is consistent with the result used by IAHP. In contrast, the evaluation results of the method proposed in this paper are more intuitive. Each evaluation index has a corresponding score, which is conducive to the precise positioning of quality improvement.

6 Conclusion

Aiming at the shortcomings of ITMS QE research, this paper constructed a new quality model based on the characteristics including intelligence, complexity and interactivity. It proposed new evaluation indexes and metric elements which are suitable for ITMS on the basis of ISO/IEC 25010. The quality model is dynamic and can be used to guide ITMS evolution. It introduced a quality evaluation method based on IAHP, which avoids the large amount of calculation and complex process in the consistency verification process. At the same time, the evaluation calculation logic was encapsulated to develop an ITMS QE prototype tool. The experiments showed that this method fully considers the generality and characteristics of ITMS. It can provide effective evaluation results. In the evaluation process, there are measures such as model customization, weight determination, and comprehensive quality analysis, which can adapt to changes in the application environment and evaluation needs. Therefore, the method has good practicability and makes the ITMS QE work standardized and systematic. In view of the limitations of the method, further research and improvement are needed. In future work, we will further refine ITMS quality requirements, software features, measurement data collection and analysis. According to the actual application effect, we hope the proposed method can improve the credibility of ITMS QE and analysis.

Footnotes

Acknowledgments

The authors would like to thank the anonymous reviewers for their insightful comments and suggestions. This work was supported by the National Natural Science Foundation of China (under No. 61872078).

Source code:

References

Changjun

, Hongtong

and Jianhua

, Intelligent Traffic Management System Theory and Practice, People’s Public Security University Press, 2015.

Jiang

, Yin

and Kaynak

, Data-driven monitoring and safetycontrol of industrial cyber-physical systems: Basics and beyond, IEEE Access 6 (2018), 47374–47384.

Evaluation (Square) – System and Software Quality Models, 2011.

ISO/IEC 25021: Systems and Software Engineering – Systems and Software Quality Requirements and Evaluation (Square) – Quality Measure Elements, 2012.

ISO/IEC 25023: Systems and software engineering–Systems and software Quality Requirements and Evaluation (SQuaRE)–Measurement of system and software product quality, 2016.

Boehm

B.W.

, Brown

J.R.

and Lipow

, Quantitative evaluation of software quality, In International Conference on Software Engineering, 1976.

Mccall

J.A.

, Richards

P.K.

and Walters

G.F.

, Factors in software quality – volume 1 – concept and definitions of software quality, Factors in Software Quality, 1977.

Kan

S.H.

, Basili

V.R.

and Shapiro

L.N.

, Software quality: Anoverview from the perspective of total quality management, Ibm Systems Journal 33(1) (1994), 4–19.

Bansiya

and Davis

C.G.

, Quantitative evaluation of softwarequality, IEEE Transactions on Software Engineering 28(1) (2002), 4–17.

10.

Kan

S.H.

, Metrics and models in software quality engineering, Software Guide, 2009.

11.

Zhang

, Li

, Wang

, Xu

and Shao

, A hierarchical model for quality evaluation of mixed source software based on iso/iec 25010, International Journal of Software Engineering and Knowledge Engineering, 2022.

12.

Wang

, Hou

and Yao

, Research on quality analysis andprediction method of smart electric energy meter in operation, Electrical Measurement and Instrumentation 11(4) (2022), 1–7.

13.

, Lin

, Nanning

, Feiyue

and Yuehe

, Artificial intelligence test:a case study of intelligent vehicles, Artificial Intelligence Review 50(3) (2018), 441–465.

14.

Wang

, Wu

, Mao

, Ding

, Gu

and Li

, Growing construction and adaptive evolution of complex software system, Science in China: Information Science 44(6) (2014), 743–761.

15.

Meng

, Product quality risk assessment of complex manufacturing system based on fuzzy petri nets, Lanzhou University of Technology, 2021.

16.

, Liu

, Wang

, Xiang

, Zhu

and Xiong

, Research onservice quality evaluation and maintenance quality control standardsystem of complex electromechanical system, Standardization inChina (06) (2020), 154–158.

17.

, Feng

and Dong

, Fuzzy comprehensive evaluation of expresswayintelligent traffic management system based on ahp, Police Technology 02 (2020), 90–94.

18.

, Li

, Wang

, Liu

and Dai

, Aquality evaluation method for software testing about safety-critical software, In 2022 IEEE 13th International Conference on Software Engineering and Service Science (ICSESS), pages 35–42, 2022.

19.

Zhang

, Sun

and Chen

, Software quality evaluation based on test data, Ship Electronic Engineering, 2019.

20.

Sun

, Lv

, Cai

, Pan

and Huang

, Software test quality evaluation based on fuzzy mathematics, Journal of Intelligent & Fuzzy Systems 40(4) (2021), 6125–6135.

21.

Awasthi

and Omrani

, A hybrid approach based on ahp and belief theory for evaluating sustainable transportation solutions, International Journal of Global Environmental Issues 9(2009), 212–226.

22.

Jiang

and Yin

, Recursive total principle component regression based fault detection and its application to vehicular cyber-physical systems, IEEE Transactions on Industrial Informatics 14(4) (2018), 1415–1423.

23.

Zervos

and Pelekanos

N.T.

, Current transport in semiconductor nanowires with built-in barriers based on a 1d transfer matrix calculation, Journal of Applied Physics 104(5) (2008).

24.

Bolstad

B.M.

, Irizarry

R.A.

, Astrand

and Speed

T.P.

, A comparison of normalization methods for high density oligonucleotide array databased on variance and bias, Bioinformatics 19(2) (2003), 185–93.

25.

Zhang

, Wei

and Yang

, Research on lightweight mvc framework based on spring mvc and mybatis, In 2013 Sixth International Symposium on Computational Intelligence and Design (ISCID) 1(2013), 350–353.

26.

Saaty

T.L.

and Kearns

K.P.

, The analytic hierarchy process, Analytical Planning, 1985.

A quality evaluation method for intelligent traffic management system based on business data flow

Abstract

Keywords

1 Introduction

2 Literature review

2.1 Quality evaluation for individual software

2.2 Quality evaluation for complex systems

2.3 Quality evaluation methods

3 ITMS quality evaluation requirements

3.1 ITMS architecture

3.2 ITMS quality requirements analysis

3.2.1 Quality requirements of intelligent systems

3.2.2 Quality requirements of complex systems

3.2.3 Quality requirements formed by the interaction and integration of multiple subsystems

4 Methodology

4.1 Determining evaluation dimensions

4.1.2 Data transmission

4.1.3 Data application

4.2 Building attribute knowledge base

5.1.2 Result analysis

5.2.1 Results

5.3 Case study #3

IAHP

AHP

Table 5 Value list of random consistency indicator RUI\label tab5 n 1 2 3 4 5 6 7 8 9 RUI 0 0 0.58 0.9 1.12 1.24 1.32 1.41 1.45

Fuzzy comprehensive method

Footnotes

Acknowledgments

References

Table 5
Value list of random consistency indicator RUI\label tab5

n 1 2 3 4 5 6 7 8 9

RUI 0 0 0.58 0.9 1.12 1.24 1.32 1.41 1.45