Evaluation for quality regulatory capability maturity in government procurement of public service based on probabilistic linguistic term sets

2.1 Evaluation of regulatory capability

Regulatory capability, as one of governments’ capabilities, mainly focuses on implementing supervising policies, and ensuring supervisory quality through executing these policies. Supervisory capability is different from supervisory performance. However, sound supervisory performance needs powerful regulatory capability as a guarantee. All factors should be comprehensively taken into consideration to evaluate the regulatory capability, such as the building of regulatory organizations, application of supervisory tools, and policy making for regulation [23]. Scientific and logical evaluating index system is necessary to make a precise assessment, so that weak links and relative gaps in the regulation system can be found out. The regulation level will be increased through implementation of effective improving measures, which are to the point. In 1995, the Organization for Economic Co-operation and Development (OECD) issued the first international standard on construction of regulation system and evaluation of regulatory capability, which made it clear that regulatory capability should composite of three ingredients: organizations, policies and tools for regulation. There are plenty of studies on composition and evaluation of regulatory capability in recent academic world. According to current literature, evaluation of regulatory capability may stick to one specific organization, or to one certain area or industry. Subjects and matters to be evaluated are generally set from their composition, including extent of resources’ abundance, relationship between subjects and objects in supervision, establishment and implementation of regulation policies, relative independence of regulation organizations. Kjekshus and Veggeland [24] suggested that supervision capability is the capability for formal regulation organizations to carry out a series of measures to ensure that the goal of supervision correspondence to a nation’s decision-making. Therefore, performance of regulation policies’ implementation should be an important constitutive of regulatory capability. Yadav [25] showed that firms used a systematic process to build capabilities for managing regulations, and offered a process model for firms to build regulatory capabilities. Mcallister [26] proposed that the extent of resources’ abundance is the base for regulatory capability evaluation. Rooij [27] suggested that the extent of resources’ abundance restricted improvement of local governments’ environmental regulatory capability, as well as the improvement of supervision performance. Liu et al. [28] established a graduated measurement indicator system on regulatory capability which includes the building of regulatory institution, the using of regulatory tools and quality of regulatory policies. The Ministry of Environmental Protection (MEP), China Food and Drug Administration (CFDA) and the Center of Inspection and Supervision of National Health and Family Planning Commission (CISNHFPC) were chosen to be measured and graduated by their proposed system. Aiming at the regulatory capability of urban public utility, Li [29] summarized the enlightenment and reference of American evaluation system of urban public utilities regulation. Yang [30] suggests learning from the idea of responsive regulation that emerged recently in western countries and creating a new paradigm of collaborative regulation between state and non-state actors. Hu [31] established a structural model of institutional capacity of drug administration as well as an appraisal index system based on the concepts of potential capacity and realized capacity by Nielsen. Ten provincial drug administrations’ regulation capacities were then evaluated and compared based on the established model. Wu [32] proposed an evaluation index system of regulatory capability including three dimensions: constitute elements of governments’ regulatory capability, explicit form and influence factors, which provides scientific standard for governments’ judgement.

In summary, there are many researches on governments’ regulatory capability, as well as its evaluation. In this field, great achievements have been made and abundant experience has been accumulated both in theoretical research and practice. However, imperfections still exist. Studies on evaluation indicators and methods hold out a situation of hundreds of flowers in blossom, which is low in consensus about these research findings, and relatively stable evaluation mode hasn’t yet been developed. For one thing, most researches in evaluation of regulatory capability try to find out key factors influencing the regulatory capability through one-time evaluation, against which measures are taken to improve the capability, while evaluations considering dynamic condition are ignored. Therefore, it is difficult to establish a dynamic improving mode due to lack of continuous improving idea. For another thing, theoretical research on regulatory capability evaluation tends to be disconnected from practice, and many evaluation methods in theoretical research is hard to attain good results from practice, which is low in promotional value.

2.2 Regulation in government procurement of public service (GPPS)

As to the regulation in GPPS, there have been plenty of relative studies. Lodi et al. [33], Johansson et al. [34] both focused on the contract design and control to better regulate the process of GPPS. Wu et al. [32] tried to establish a relatively complete regulatory system of GPPS, and took a close look at the role transition of government in GPPS, thus underlining government’s supervision and regulation role. Cunningham and James [35] analyzed GPPS from the value of a regulatory perspective. They revealed that a ‘soft’ regulatory framework that offered little support to partnership relations between voluntary organizations and local authorities. Ainuddin et al. [36] indicated that the principle of transparency in procurement of goods and government services was important to be accommodated for the purpose of better regulation. Maciejewski [37] discussed some applications of big data methods in public administration, which also drew attention of big data application in regulation in GPPS. Ida and Talit [38] looked into structural reforms in the regulation of public bus services in Israel. They studied and examined the reasons and effectiveness of different measures taken to regulate outsourced the public service.

Academic research in the field of regulation and management in GPPS has mainly dealt with regulation mechanisms, risks, systems and the implementation of governments’ regulation responsibilities, etc. However, little research has been done in respect to evaluation of the regulatory capability in GPPS, as well as the capability maturity. At present, China is still in primary stage in GPPS, the overall level of which is rather low. In present practice, restricted by many factors such as ideological understanding, human resource, technological equipment, legal basis, etc., regulation in GPPS is still a weak link which hasn’t received adequate attention.

3.1 CMM and the feasibility analysis of its application into quality regulatory capability evaluation criteria development

Proposed by Carnegie Mellon University, CMM was initially applied to deal with software development and management process. It is a description on each stage for defining, implementing, measuring, controlling and improving the software’s process. The initiator, Institute of Software Engineering of Carnegie Mellon University, divided the capability maturity of software development into 5 levels, 18 key process areas, 52 objectives and 316 key practices. CMM made it clearer and more convenient to identify the capability of present process, as well as the important problems of software quality and process improvement, thus providing guidelines to make decisions on selecting process improvement strategies. As software development itself is a project management process, it is undoubtedly logical to apply CMM into the field of project management. After that, CMM was successively applied into areas such as knowledge management [39], Risk management [40], service management [41, 42], public management and logistics management [43]. CMM has now become a relatively mature tool to define the stage of development, characteristics of stages and the direction of development. The five-level maturity model for project management proposed by Kerzner includes general terms, general processes, single method, baseline comparison and continual improvement. Each level has specific evaluation method and criterion, so that the maturity degree of this level could be obtained by aggregation, based on which deficiencies are then analyzed and improving measures are thus made to get into the next stage. Krishnan et al. [44] used CMM to assess the process maturity of technology projects management in the US federal government. Pongsuwan [45] proposed a measurement model called “Procurement Competitive Capability Maturity” (PCCM) for a company to assess current practices of procurement function and perceive the level of its capabilities. Rendon [46] presented an assessment of contract management process maturity in the US Navy based on CMM, which is used to benchmark an organization’s contract management process maturity.

Quality supervision and regulation of GPPS is a system engineering, which develops from disorder to order, from passive to active supervision, from fragmented to systematic. Therefore, it is a process which is gradually getting maturity and the comprehensive capability of management is gradually improving. Meanwhile, the core idea of CMM is just to find out and work on the disadvantages through evaluating the process of an organization and its capability maturity level, thus forming a continuous improving mode. Applying CMM into criteria development of quality regulatory capability evaluation in GPPS is feasible and reasonable. For one thing, CMM is particularly suitable for evaluating process or management capability, and here what we want to assess is just the regulatory capability in GPPS. The weak links in GPPS can be found out through maturity evaluation, which provides direction and effective guidance for departments concerned to make out appropriate improving measures and to perfect the quality supervision system. For another thing, it can provide references for senior governments and the public to evaluate local governments’ performance in their procurement of public service, as well as their supervision performance, thus urging local governments continuously improving their quality regulatory capability in the process of promoting the practice of GPPS. Therefore, it is not only feasible, but also very necessary to introduce CMM theory into evaluation and enhancement of quality regulatory capability of GPPS. Through determining the maturity level of local governments’ quality regulatory capability in its procurement of public service, and defining the weak links facing process areas, aiming at which improvements are made and capability is developed stage by stage. Governments’ regulatory capability in their procurement of public service is consequently enhanced step-by-step, and effective improving mode is thus developed.

3.2 Evaluation criteria development of regulatory capability maturity in GPPS

3.2.2 Framework of quality regulatory capability maturity evaluation in GPPS

The proposed framework of quality regulatory capability maturity evaluation in GPPS is described as follows (Fig. 1):

Developing evaluation criteria, which means determining the critical indexes to evaluate the quality regulatory capability maturity on basis of the key links in quality supervision in GPPS.

Identifying maturity levels, which means dividing the capability governments should have to regulate the quality of their procurement of public service into different levels, as well as the characteristics of these levels.

Implementing the evaluation, which means that various experts (selected as assessors) make a judgement based on the developed criteria and characteristics of different levels. The judgement is about which level the evaluated object currently belongs to corresponding to each evaluation index in the criteria.

Aggregation of different experts’ judgements. An expertise aggregation method is designed to fully reflect the experts’ preferences and evaluation information.

Determining the order for improvement, which means that in view of the advantages and disadvantages in the process of quality supervision of GPPS, and according to the evaluation results, specific direction for next work for improvement is determined, and the capability enhancement is implemented with emphasis.

OPEN IN VIEWER

Fig.1

Flow chart of quality regulatory capability maturity evaluation in GPPS.

4.1 Preliminaries

Firstly, some basic definitions and operations for PLTSs are reviewed in this section.

Definition 1. (Pang et al., 2016) Let S = {s₀, s₁, ⋯ , s _τ } be a LTS, a PLTS is defined as

L ( p ) = { L ( k ) ( p ( k ) ) | L ( k ) ∈ S , p ( k ) ≥ 0 , k = 1 , 2 , ⋯ , # L ( p ) , ∑ k = 1 # L ( p ) p ( k ) ≤ 1 }(1)

Where # L (p) is the number of all different linguistic terms in L (p).

From the Def. 1, we can easily observe that the sum of probabilistic distribution of all possible linguistic terms is not more than 1. If ∑ k = 1 # L ( p ) p ( k ) < 1 , then some information is not embodied owing to lack of knowledge or hesitance for DMs. To overcome this issue, Pang et al. (2016) proposed a normalization method for PLTS.

Definition 2. Let L (p) = {L^(k) (p^(k)) |L^(k) ∈ S, p^(k) ≥ 0 , k = 1 , 2 , ⋯ , # L ( p ) , ∑ k = 1 # L ( p ) p ( k ) < 1 } be a PLTS, the normalized PLTS can be defined as

L · ( p ) = { L ( k ) ( p ( k ) . ) | L ( k ) ∈ S , p ( k ) . ≥ 0 , k = 1 , 2 , ⋯ , # L ( p ) , ∑ k = 1 # L ( p ) p ( k ) . = 1 }(2) where p ( k ) . = p ( k ) / ∑ k = 1 # L ( p ) p ( k ) for k = {1, 2, ⋯ , # L (p)}.

In order to compare different PLTSs, the score function and deviation function are defined by Pang et al. (2016) as follows.

Definition 3. Let L (p) = {L^(k) (p^(k)) |k = 1, 2, ⋯ , # L (p)} be a PLTS, then the score function E and Δ for PLTs can be defined as E ( L ( p ) ) = s α(3) Δ ( L ( p ) = ( ∑ k = 1 # L ( p ) ( p ( k ) ( r ( k ) - α ) 2 ) 1 / 2 ) / ∑ k = 1 # L ( p ) p ( k )(4)

Where α = ∑ k = 1 # L ( p ) ( p ( k ) r ( k ) ) / ∑ k = 1 # L ( p ) p ( k ) , and r^(k) is the subscript of linguistic term L^(k)(k = 1, 2, ⋯ , # L (p)).

Given arbitrary two PLTSs L₁ (p) and L₂ (p), then the following comparative conclusions hold:

if E (L₁ (p)) < E (L₂ (p)), then L₁ (p) ≺ L₂ (p);

Distance measure is very important and necessary in solving MCDM problems. The Euclidean distance for PLTS proposed by Pang et al. (2016) can be described as follows:

Let L 1 ( p ) = { L 1 ( k ) ( p 1 ( k ) ) | k = 1 , 2 , ⋯ , # L 1 ( p ) } and L 2 ( p ) = { L 2 ( k ) ( p 2 ( k ) ) | k = 1 , 2 , ⋯ , # L 2 ( p ) } be two arbitrary PLTSs and # L₁ (p) = # L₂ (p), then the Euclidean distance between L₁ (p) and L₂ (p) can be calculated by d ( L 1 ( p ) , L 2 ( p ) ) = ∑ k = 1 # L 1 ( p ) ( p 1 ( k ) r 1 ( k ) - p 2 ( k ) r 2 ( k ) ) 2 / # L 1 ( p )(5) where r 1 ( k ) and r 2 ( k ) are the subscripts for L 1 ( k ) and L 2 ( k ) , respectively.

Remark 1. If # L₁ (p) < # L₂ (p), then add corresponding elements for # L₁ (p) until # L₁ (p) = # L₂ (p).

Given an evaluation problem for quality regulatory capability maturity in GPPS, we assume that there are m criteria C₁, C₂, ⋯ , C _m , n alternatives A₁, A₂, ⋯ , A _n to be evaluated, and experts provide evaluated value by linguistic terms for every alternative in terms of all criteria. Based on processing the data, a decision-making matrix B = (b _ij ) _m×n was obtained, where b _ij is a PLTS indicating the value for alternative A _i with respect to criterion C _j . The criteria weighting vector w = (w₁, w₂, ⋯ , w _n ) is completely unknown satisfying ∑ j = 1 n w j = 1 , w _j  ≥ 0.

5.1 Description

The quality regulatory capability evaluation for GPPS in Jiangsu Province of China is taken as an example here to illustrate the specific evaluation procedure. Six typical cities in Jiangsu are selected to be evaluated. They are Nanjing (A₁), Wuxi (A₂), Suzhou (A₃), Nantong (A₄), Lianyungang (A₅) and Xuzhou (A₆), among which two (Suzhou and Wuxi) are from South of Jiangsu, two (Nanjing and Nantong) Middle of Jiangsu, and two (Xuzhou and Lianyungang) the Northern part. Therefore, these 6 cities are representative of the current condition of quality regulation in Jiangsu Province.

5.2 Evaluations

(1) Building an evaluation group of experts

To ensure the reliability and accuracy of the evaluation, our evaluation group is composed of 10 experts in the field of public management from different universities such as Nanjing University, Nanjing University of Aeronautics and Astronautics, Yangzhou University, Jiangsu University. Officials from governments concerned who are in charge of the work of procurement of public service and personnel from service suppliers are also included in our expert group. The whole group constitutes 10 experts.

(2) Collecting materials for evaluation

Materials for references involved in evaluating quality supervisory capability maturity in government procurement of public can be classified into three categories. The first category is materials about construction condition of government concerned, such as the establishment of governments’ regulation and coordination agency and special regulating organization, as well as the allocation and training of staffs, documents of policies and relative institutions, and so on. The second category is materials collected through searching on the internet such as policy information about GPPS in the 6 evaluated cities in Jiangsu, the bidding information, information from platforms of governments’ work on internet, etc. The third category is typical examples and cases about quality regulation in GPPS in these 6 cities provided by local governments or through search from the internet. All of the above information considered, initial questionnaire is designed according to different key process areas and their goals, which are delivered to all members in our evaluation group through internet, as well as all the materials for references.

(3) The expert group making a judgement (evaluation)

Members of the evaluation group go over all relative material received carefully, and they can communicate with each other anytime on line to gain further insight or discuss with each other when they have different opinions. Finally, evaluation information is collected from experts. All members in our group should make a judgment in view of each cities’ current maturity level out of the five levels (L1-L5) in CMM corresponding to the five criteria (C₁, C₂, C₃, C₄, C₅).

When judgments from each expert are all collected (Table 2), the evaluation results are sorted out and analyzed. Here, A₁, A₂, A₃, A₄, A₅, A₆ represent the six cities respectively: Nanjing, Wuxi, Suzhou, Nantong, Lianyungang and Xuzhou.

(4) Aggregation of experts’ evaluation

According to the definition of PLTS, Table 2 can be translated into the following decision matrix B with PLTSs. Here, we assume that the five levels (L1, L2, L3, L4, L5) are the corresponding linguistic fuzzy terms s₀, s₁, s₂, s₃, s₄. The probability for the linguistic fuzzy term is calculated by ratio of the members of experts choosing this level out of the total members of experts. B = [ { ( s 0 , 0.1 ) , ( s 1 , 0.6 ) , ( s 2 , 0.3 ) } { ( s 1 , 0.6 ) , ( s 2 , 0.3 ) , ( s 3 , 0.1 ) } { ( s 0 , 0.1 ) , ( s 1 , 0.3 ) , ( s 2 , 0.5 ) , ( s 3 , 0.1 ) } { ( s 1 , 0.5 ) , ( s 2 , 0.4 ) , ( s 3 , 0.1 ) } { ( s 1 , 0.3 ) , ( s 2 , 0.5 ) , ( s 3 , 0.2 ) } { ( s 1 , 0.6 ) , ( s 2 , 0.2 ) , ( s 3 , 0.2 ) } { ( s 0 , 0.2 ) , ( s 1 , 0.6 ) , ( s 2 , 0.2 ) } { ( s 0 , 0.1 ) , ( s 1 , 0.6 ) , ( s 2 , 0.3 ) } { ( s 0 , 0.3 ) , ( s 1 , 0.6 ) , ( s 2 , 0.1 ) } { ( s 0 , 0.1 ) , ( s 1 , 0.7 ) , ( s 2 , 0.2 ) } { ( s 0 , 0.2 ) , ( s 1 , 0.4 ) , ( s 2 , 0.4 ) } { ( s 0 , 0.1 ) , ( s 1 , 0.6 ) , ( s 2 , 0.3 ) } → { ( s 0 , 0.1 ) , ( s 1 , 0.8 ) , ( s 2 , 0.1 ) } { ( s 1 , 0.3 ) , ( s 2 , 0.4 ) , ( s 3 , 0.3 ) } { ( s 1 , 0.5 ) , ( s 2 , 0.5 ) } { ( s 0 , 0.2 ) , ( s 1 , 0.6 ) , ( s 2 , 0.2 ) } { ( s 0 , 0.1 ) , ( s 1 , 0.4 ) , ( s 2 , 0.4 ) , ( s 3 , 0.1 ) } { ( s 0 , 0.3 ) , ( s 1 , 0.5 ) , ( s 2 , 0.1 ) , ( s 3 , 0.1 ) } { ( s 1 , 0.7 ) , ( s 2 , 0.3 ) } { ( s 1 , 0.5 ) , ( s 2 , 0.3 ) , ( s 3 , 0.2 ) } { ( s 1 , 0.2 ) , ( s 2 , 0.5 ) , ( s 3 , 0.2 ) , ( s 4 , 0.1 ) } { ( s 0 , 0.2 ) , ( s 1 , 0.6 ) , ( s 2 , 0.2 ) } { ( s 1 , 0.6 ) , ( s 2 , 0.3 ) , ( s 3 , 0.1 ) } { ( s 1 , 0.4 ) , ( s 2 , 0.6 ) } { ( s 0 , 0.2 ) , ( s 1 , 0.6 ) , ( s 2 , 0.2 ) } { ( s 0 , 0.2 ) , ( s 1 , 0.5 ) , ( s 2 , 0.3 ) } { ( s 0 , 0.1 ) , ( s 1 , 0.4 ) , ( s 2 , 0.5 ) } { ( s 0 , 0.3 ) , ( s 1 , 0.7 ) } { ( s 0 , 0.2 ) , ( s 1 , 0.3 ) , ( s 2 , 0.5 ) } { ( s 0 , 0.1 ) , ( s 1 , 0.3 ) , ( s 2 , 0.6 ) } ]

Next, quality regulatory capability maturity in GPPS of the six cities will be evaluated and ranked based on the TODIM method for PLTSs.

Step 1. Calculate the criteria weights w = (0.18, 0.2, 0.12, 0.21, 0.28) according to Equation (7)

Step 2. Compute the relative weights w _r  = (0.63, 0.69, 0.44, 0.76, 1) based on Equation (8).

Step 3. Calculate the dominance degree for A _i over A _k in all criteria Φ (A _i , A _k ) according to Equations (9-10) (shown in Table 3).

Step 4. Compute the overall dominance degree Z ( A 1 ) = 0.6 , Z ( A 2 ) = 0.82 , Z ( A 3 ) = 1 , Z ( A 4 ) = 0.38 , Z ( A 5 ) = 0.06 , Z ( A 6 ) = 0

Therefore, the ranking result is A₃ ≻ A₂ ≻ A₁ ≻ A₄ ≻ A₅ ≻ A₆.

To illustrate the advantage and feasibility of our method, we carry out a comparative analysis with TOPSIS method proposed by Pang et al. [16], in which the main decision-making procedure is shown as follows:

We apply the above procedure to solve our decision problem as follows:

The criteria weights are w = (0.18, 0.2, 0.12, 0.21, 0.28);

5.4 Comparison with VIKOR method proposed by Wang et al. (2018)

The main decision steps of VIKOR method proposed by Wang et al. (2018) can be described as follows.

Calculate the criteria weights w = (w₁, w₂, ⋯ , w _n ) ;

We use the VIKOR method to solve our decision problem as follows.

To make the comparison reasonable we choose the weights w = (0.18, 0.2, 0.12, 0.21, 0.28) ;

It can be easily found that the results are different between our proposed method and the two existing methods, as shown in Table 4. The main reason lies in the fact that our proposed method uses TODIM considering the bounded rationality of decision makers. In fact, in most cases, decision makers may be bounded rational and focus more on changes between the reference point and the actual evaluation. Therefore, our method may be more reasonable than the above existing methods.

From the results in Table 3, we can see that Suzhou ranks first among these 6 cities, while Xuzhou ranks the 6th. In fact, as a developed city not only in Jiangsu, but also in China, Suzhou also has done very well in the implementation of the policy of GPPS. Suzhou has published different kinds of guidelines and measures to implement and promote the policy of GPPS. “Guidelines for implementation of GPPS”, “Methods for cost regulation in GPPS”, “Rules for the implementation of GPPS”, et al. have been issued in succession recent years, which contribute a lot to the successful promotion and effective management of GPPS in Suzhou. Meanwhile, there’re many innovations and good practices of GPPS in Suzhou, and plenty of experiences related to the quality supervision and regulation have also been gained. Suzhou excels not only in policy making, but also in process management, multi-part participation, as well as in the effectiveness. So, it is quite reasonable and understandable that Suzhou ranks the first. Then, we can see from Table 12 that Nanjing ranks the 3rd among these 6 cities. Nanjing is the provincial capital of Jiangsu. It should have done well in GPPS in most peoples’ opinions. As a matter of fact, it indeed has done well in certain aspects, especially in policy making and implementation, as well as in multi-party participation. As shown in Table 3, in policy making and implementation, Nanjing overmatches the other 5 cities, ranking the first. In multi-participation, Nanjing ranks 2nd, only slightly weaker than Suzhou. In order to enhance the quality regulation capability in GPPS, Nanjing should do much more in process management, and focus on the improvement of effectiveness. We also take a close look at Lianyungang and Xuzhou, which rank the 5th and 6th among the 6 cities. Although Lianyungang is better than Xuzhou in this ranking list, the advantage is rather minor. The two are almost the similar in quality regulation in GPPS. Much remains to do for them to enhance their capability. Effective policies needed to be made and issued to guide the practice, more attention should be paid to multi-party participation of the quality regulation, and they should also learn from others to do more efficiently in process management, so as to improve the effectiveness at last.