Cloud service recommendation for small and medium-sized enterprises: A context-aware group decision making approach

Abstract

Recently, the enormous advantages of cloud services make them increasingly appealing to the small and medium-sized enterprises. The growing number of available services makes it challenging to select trustworthy services. Existing approaches focus on user preferences to guide personalized services recommendation for individual users, but lack of the research on trustworthy service recommendation for the small and medium-sized enterprises that represents a group user consisting of multiple individual users. For this type of enterprise, the cloud services recommendation must address the challenges from the diverse client context of individual users, the imprecise quality of experience in an uncertain cloud environment and the invalid or unsatisfactory recommendations. A client context-aware approach is proposed to recommend trustworthy cloud services for the small and medium-sized enterprises based on non-compensatory multi-criteria decision-making. In it, a type of client context is viewed as an independent evaluation criterion, and the interval neutrosophic numbers are employed to measure the fuzzy trustworthiness of cloud services. Based on the investigated outranking relations of interval neutrosophic numbers, a non-compensatory multi-criteria decision-making procedure via an improved ELECTRE III method is developed to rank candidate services. Experimental results demonstrate that this approach could efficiently produces the accurate ranking results of cloud services and effectively recommend the trustworthy service for small and medium-sized enterprises.

Keywords

Client context cloud service recommendation small and medium-sized enterprise ELECTRE III interval neutrosophic sets

1 Introduction

1.1 Motivation

Currently, the business activities of enterprises have been generating the massive data [1]. These data could be analyzed and are of great significance to improve the enterprise operations and management decisions. As the infrastructure of big data, cloud services (CSs) have been widely recognized by users. The investigation of VMWare shows that twenty-seven per cent of information technology (IT) work load is executed in cloud computing platform in 2016, and predicts that eighty-one per cent of IT work load will run in clouds in the future.

An enterprise is considered a small and medium enterprise (SMEs) when it has less than 250 employees [2]. SMEs are an important part of national economic systems. According to the report [3], there are more than 20 million registered SMEs in China at the end of 2015. In France, 99.9% of enterprises in the construction sector are SMEs [2]. Due to the limited budget, an SME usually cannot bear the expensive cost of building an private cloud, and it has become a prevailing trend to exploit the economic and effective public CSs to meet the urgent demands of SMEs for the storage, analysis and application of big data [4 –6]. However, large number of candidate CS offerings, with the same or similar functions, have created a hot topic among researchers as they seek methods to help users select highly trustworthy CSs [7].

The existing approaches mainly employ the user preferences to achieve personalized CS recommendation for individual users [8 –12]. In contrast to an individual user, an SME represents a group user consisting of multiple individual users. Furthermore, different from the large-scale enterprises, the SMEs’ client context is more diverse when they invoke CSs via different access devices and networks, and the CS’s quality of service (QoS) experienced by SMEs is more uncertain because of the dynamic work load in clouds. Thus, the CS recommendation for an SME must deal with the following challenges:

The diversity of client context makes it complex to select the highly trustworthy CSs for an SME. As we known, the performance of CSs is usually dynamic due to the susceptibility and vulnerability of a cloud environment [9, 13], and the variation of user loads leads to changes in the CSs’ performance over time [10, 14]. More importantly, the existing literature indicates that the quality of experience (QoE) of users is affected by the client context, including the client device [15], locations [16], autonomous systems (ASs) [17]. The individual users from an SME may experience different QoS of one CS in a different client context at the same time. Thus, to find the trustworthy CSs for an SME, the decision maker must attach importance to the context’s effects on QoE and comprehensively evaluate the QoS of CSs from the angle of a group user.

The compensable multi-criteria decision-making (MCDM) approaches are unable to filter out the invalid or unsatisfactory candidate CSs. The CS recommendation is usually modelled as an MCDM problem [9 –11], in which the evaluation data from multiple criteria is aggregated with the weights. The multiple QoS attributes (e.g., usage cost, fault rate, response time, operability, privacy and availability [18]), multiple experts with different confidence level and risk attitudes [11], multiple sources of information that provide QoS evaluations [17], and multiple time periods [9, 10] may become the decision criteria for CS selection. These MCDM-based approaches assume that the decision criteria are compensable, but it is not the case for an SME. Under this assumption, the MCDM of CS recommendation might select an invalid or unsatisfactory candidate as the optimal CS for an SME based on fuzzy evaluation data, as shown in Section 3.

To measure the uncertain QoS of CSs based on the fuzzy evaluation data, some theories, such as the probability theory [19], fuzzy mathematics [20], rough set [21], interval number [22], evidence theory [17], are utilized in the existing researches. By introducing the interval neutrosophic set (INS) theory [23], two MCDM-based CS recommendation approaches [9, 10] are proposed by us and demonstrate the good performance in measuring the uncertainty of CSs’ QoS for individual users. However, these approaches supporting the compensation of multiple criteria neither identify the tolerable/unbearable threshold nor distinguish the minor/major difference for sensitive users. Thus, they cannot eliminate the possibility of selecting an invalid or unsatisfactory candidate as the optimal one for an SME.

The ELECTRE III [24, 25], in contrast to other MCDM procedures, is a non-compensatory method, which implies that a poor evaluation on a single criterion is not compensated for high scores in other criteria. This feature makes it more appropriate to solve the problem mentioned above, since an SME tends to discard a candidate that is proven poor on one criterion comparing to another candidate. However, the traditional ELECTRE III is quite difficult to apply directly into the CS recommendation for an SME due to deficiencies [26], such as the limited type of data not supporting the fuzzy evaluations, the complicated process of ranking computation and the partial order of all alternatives. Especially, the traditional ELECTRE III requires the users to provide exact threshold values for ensuring the correct ranking result of candidates, but this is not practical.

1.2 Our contributions

Aiming at the above challenges and the limitations of the existing researches, we formulates the SMEs-oriented CS recommendation problem as a non-compensatory MCDM problem via INS theory in order to select a highly trustworthy CS for an SME. In this problem, a type of client context is viewed as an independent evaluation criterion and the interval neutrosophic numbers (INNs) are employed to measure the fuzzy trustworthiness of CSs. An improved ELECTRE III method supporting INNs is developed to solve the non-compensable MCDM problem. Our contributions are as follows:

An improved ELECTRE III method with INS is proposed to filter out the invalid or unsatisfactory candidate CSs. To make it feasible to solve the SMEs-oriented CS recommendation problem, the improved method employs an INS to evaluate the fuzzy trustworthiness of a CS, and simplifies the process of ranking computation for obtaining a total order of all candidate CSs. Meanwhile, the preference, indifference, and veto thresholds of the decision criteria are redefined with the relative ratio values rather than absolute values. The relative ratio thresholds are adaptive to enable reducing the difficulty of providing the exact values for users. These improvements facilitate the increase of the accuracy of SMEs-oriented CS recommendation.

The outranking relations of INNs are proposed for supporting the new calculations related to the improved ELECTRE III method. We employ the INNs to assess the fuzzy trustworthiness of CSs from the perspective of client context analysis. A type of client context is viewed as an independent evaluation criterion, and the trustworthiness of CS is measured from three aspects including the performance, uncertainty, and potential risks. To conveniently support the non-compensatory MCDM of SMEs-oriented CS recommendation, the outranking relations of INNs based on an improved ELECTRE III method are developed to calculate the ranking values of candidate CSs based on theoretical proofs.

We examine the proposed approach through experiments using a real-world dataset. The results demonstrate that the improved ELECTRE III method via INS can overcome the defects of the traditional method applied to the CS recommendation, and the proposed non-compensable MCDM procedure is effective for solving the SMEs-oriented CS recommendation problem.

The remainder of the paper is organized as follows: Section II introduces the related work. Section III gives the problem statement. Section IV defines the outranking relations of INNs for an improved ELECTRE III method. Section V puts forward the non-compensatory MCDM procedure of SMEs-oriented CS recommendation. Section VI analyzes the experimental results. Finally, Section VII concludes the paper.

2 Related work

2.1 MCDM and ELECTRE method

MCDM attempts to handle the real-life decision problems, involving several conflicting criteria that should be taken into account conjointly and a set of alternatives to be chosen from, in order to reach at a reasonable decision [24]. The majority of the existing methods, such as techniques for order preference by similarity to an ideal solution (TOPSIS), analytic hierarchy process (AHP), fuzzy analytic hierarchy process (FAHP), and analytic network process (ANP), are function models to solve MCDM problems. In these methods, the priority of the alternatives will be obtained after being aggregated by given weights and it is assumed that the criteria are compensatory. For example, even if alternative A is worse than alternative B on one criterion (A ≠ B), A may be compensated from other criteria and the final consequence may be A > B. In fact, in some cases, one criterion should not be compensated for others, such as the Example 1. If a comprehensive evaluation is still aggregated by function models in these situations, the actual will of decision makers will be ignored, which may lead to information loss. Thus, another method, namely the relation model, is proposed to cope with these situations. The relation model employs the pair-wise comparison of alternatives for each single criterion to rank the alternatives in terms of the priority order among the criteria [27]. The ELECTRE (elimination and choice expressing reality) methods are the representatives in this domain.

ELECTRE refers to one family of MCDM methods known as “outranking relations” to rank a set of alternatives [28]. Recently, the theoretical foundations of ELECTRE have been intensively investigated and different versions of the ELECTRE method were proposed, such as ELECTRE I, ELECTRE Iv, ELECTRE II, ELECTRE III, ELECTRE IV, ELECTRE TRI, ELECTRE IS, ELECTRE TRI-C and ELECTRE TRI-nC for dealing with different decision aiding situations [24]. Obviously, each of the ELECTRE versions differs operationally, and they also differ with respect to the type of problems they can be used for. ELECTRE II, III, and IV were designed for the task of constructing an ordering of the alternatives from the best to the worst, namely the ranking problem. ELECTRE II is based on true criteria, whereas the other two methods use pseudo-criteria. The concept of pseudo-criteria allows to model imperfect knowledge due to preference and indifference thresholds. The major difference between ELECTRE III and ELECTRE IV is that the latter method does not use criteria weights. ELECTRE IV is used when quantifications are not possible, whereas ELECTRE III is suitable when it is possible and desirable to quantify the relative importance of criteria [29].

However, some inadequacies [25] of the traditional ELECTRE III method make it difficult to apply directly to CS recommendation. These inadequacies include the limited types of data not supporting the fuzzy evaluations, the complicated process of ranking computation, and the partial order of all alternatives. In addition, the traditional method requires the users to provide the exact threshold values, which is impractical for the decision makers of CS recommendation in an uncertain cloud environment. Thus, although ELECTRE III as a non-compensatory method is promising to solve the CS recommendation problem for an SME, some significant improvements are indispensable.

2.2 Service selection based on MCDM

The common MCDM methods (e.g., AHP, ANP, FAHP, ELECTRE, and TOPSIS) have been used recently to solve the CS recommendation problem.

Referring to AHP and ANP, Choi et al. [18] apply a pairwise comparison matrix and an eigenvector of the matrix to calculate the weights of QoS attributes, and propose a quality evaluation system for service selection according to the QoS preferences of users. Sun et al. [11] present an AHP approach to calculate the semantic similarity between concepts, and propose an MCDM technique based on TOPSIS to rank CSs. Garg et al. [29] employ an AHP method to measure the QoS attributes and rank CSs. Similarly, Menzel et al. [31] introduce an ANP method for selecting IaaS services. Ma et al. propose a trustworthy CS selection approach that employs FAHP method to calculate the weights of user features [17] and a variation-aware CS selection approach that exploits the cloud model theory to quantify the variation characteristics of CSs’ QoS from four aspects [8]. Liu et al. [30] put forward a multi-attribute group decision-making approach for solving cloud vendor selection by integrating an improved TOPSIS. Based on the QoS time series analysis, Ma et al. [9] introduce the INS theory into measuring the trustworthiness of CSs, and formulate a time-aware CS selection as an MCDM problem, which is solved by developing an improved TOPSIS method with INS.

These approaches mainly exploit the user preferences based on historical data to achieve the personalized recommendation for individual users. There is little work undertaken for the SMEs-oriented CS selection. An SME represents a group user consisting of multiple individual users. Taking into account the characteristics of the uncertain cloud environment and the diverse user features [15 –17], it will be a feasible idea to recommend a highly trustworthy CS to an SME by modeling the different types of client contexts as the decision criteria.

2.3 NS and INS theory

Since the fuzzy set (FS) theory is presented, many extensions have been proposed to settle the issues surrounding the imprecise or uncertain information. These extensions [31] include interval-valued FS (IVFS), intuitionistic FS (IFS), and interval-valued intuitionistic FS (IVIFS). Based on the fact that IFSs cannot handle indeterminate information [32], Smarandache [33, 34] proposed the neutrosophic set (NS). An NS is a set of neutrosophic numbers (NNs) and an extension to IFS’s standard interval [0, 1]. Each NN possesses the degrees of truth, indeterminacy, and falsity, whose values lie in the non-standard unit interval ] 0^-, 1⁺ [, and the degrees of truth, indeterminacy, and falsity are independent.

For the convenience of application of NS in a practical application, Smarandache [35] and Wang et al. [23] propose an instance of NS called a single-valued NS (SVNS). In turn, Ye [36], puts forward a simplified NS (SNS), which can be described by three real numbers in the real unit interval [0, 1]. Sometimes the degrees of truth, falsity, and indeterminacy in a certain statement cannot be precisely defined in real situations, but they can be denoted by several possible interval values, requiring the interval NS. Wang et al. [23] propose the INS and interval NN (INN), and provide its set-theoretic operators. Recently, NS and its extensions have been applied into many fields, such as the decision support [37], valuation methods [38], recommender systems [39], and e-commerce [40].

NS has also been applied to the MCDM problems. Ye [32] develops an MCDM approach by using SVNS correlation coefficient measurement. Zhang et al. [41] present a correlation coefficient measure of INS and develop an MCDM method that takes into account the influence of the evaluations’ uncertainty and both the objective and subjective weights. Liu et al. [42] propose several SVNS aggregation operators based on Hamacher operations and develop an MCDM approach. Karaaslan [43] proposes two MCDM approaches under both single-valued neutrosophic refined and interval neutrosophic refined environments. Peng et al. [44] investigate the MCDM problems with multi-valued neutrosophic information.

The above work focuses on the application of NS and INS in generalized MCDM problems. To effectively address the specific problems, it is necessary to constantly develop the NS and INS theories for meeting the users’ requirements in different application scenarios. In our previous research [10], we introduce the INS theory to improve the traditional ELECTRE I method [45] and develop an MCDM-based CS ranking prediction approach for individual users. However, this approach focuses on employing INS to identify neighbouring users, not involving the outranking relations of INNs. Besides, entirely different from the ELECTRE III, the ELECTRE I neither identifies the tolerable/ unbearable threshold nor distinguishes the minor/major difference for the sensitive users. Thus, it cannot eliminate the possibility of selecting the invalid or unsatisfactory candidate as the highly trustworthy CS for an SME.

To the best of our knowledge, no similar research has investigated the SMEs-oriented trustworthy CS recommendation using ELECTRE III method with the INS theory as we have done in this paper. In our work, from the perspective of client context analysis, the highly trustworthy CS is recommended to an SME based on a non-compensatory MCDM procedure, with consideration of removing the invalid and unsatisfactory candidates.

3 Problem statement

In this section, we clarify the SMEs-oriented trustworthy CS recommendation problem with an example.

Example 1: One SME has a headquarters in the United States and a branch in the United Kingdom. To save cost, the SME is about to purchase a public CS for deploying an office automation system. The desktop computer, palmtop, and cellular phone are the most common types of client devices [15]. Assume that there are 3 candidate CSs which could provide the function matching with the requirements of the SME. To select the most suitable one for the SME, the users from the headquarters and the branch tested the 3 CSs, as shown in Fig. 1.

Fig. 1

An example of CS recommendation.

A type of client device in different locations is viewed as an independent evaluation criterion. There are six criteria, denoted by criteria #1-#6. The response time of 3 candidate CSs, denoted by CS₁-CS₃, is listed in Table 1.

Table 1

Response time data in different client context

	Headquarters			Branch
	Desktop	Palmtop	Cellphone	Desktop	Palmtop	Cellphone
	(criterion #1)	(criterion #2)	(criterion #3)	(criterion #4)	(criterion #5)	(criterion #6)
CS₁	1.46 s	1.64 s	1.38 s	1.38 s	1.46 s	2.14 s
CS₂	1.15 s	1.20 s	1.18 s	1.14 s	1.20 s	3.50 s
CS₃	1.62 s	1.82 s	1.52 s	1.51 s	1.60 s	1.41 s

Based on the ordered weighted averaging (OWA) operator, the aggregated response time data of 3 CSs, in order, is 1.577, 1.562 and 1.580 seconds. Thus, the ranking result is CS₂≻ CS₁≻ CS₃, namely, CS₂ is optimal. However, this ranking is obviously fraught with defects due to the following reasons:

Based on the assumption that the criteria are compensable, the MCDM of CS recommendation may identify an invalid candidate as the best one for an SME. Even if a CS behaves badly on one criterion, this CS is still likely to acquire an absolute advantage over all other candidates by accumulating its weak advantages on other criteria. In Table 1, although CS₂ is remarkably underperforming compared to CS₁ and CS₃ on criterion #6, CS₂ is still chosen as the optimal one due to its advantages on criteria #1-#5. However, according to “2–5–10” principles [9, 17] of response time in software testing analysis, the response time less than 2 seconds is satisfactory for users. The degree of user satisfaction will significantly decrease when the response time is more than 2 seconds, and the users usually will not accept a response time over 10 seconds. Thus, CS₂ may be an invalid candidate for some sensitive users in reality even if its overall performance is the best among all candidates because its response time substantially exceeds 2 seconds on criterion #6 while other 2 CSs do well. Unfortunately, the existing MCDM-based CS recommendation approaches enable the compensability of multiple evaluation criteria, and do not identify a tolerable or unbearable threshold that may have seriously impact on QoE. Therefore, they cannot avoid recommending an invalid candidate CS to an SME.

The compensable MCDM methods might mistakenly choose an unsatisfactory candidate as the optimal CS for an SME. In Table 1, CS₁ only retains a slight advantage that its response time is 10 percent less than that of CS₃ on criteria #1-#5. In practice, this weak advantage is often overlooked because the evaluations of the CSs’ performance are imprecise in an uncertain cloud environment. However, the response time of CS₁ is 50 percent more than that of CS₃ on criteria #6, that is, CS₁ is 50% worse than CS₃ on criteria #6. In this case, CS₃ obviously is superior to CS₁. This conclusion conflicts with the result from the comprehensive assessment that CS₁ outperforms CS₃. The reason is that the existing methods do not distinguish the minor difference and major difference among the performance evaluations, and cannot adequately handle the effects of such differences on ranking results when comparing one CS with another.

As shown in Table 1, the single value cannot exactly depict the performance variation of CSs, some approaches based on the probability theory [19], fuzzy mathematics [20], rough set [21], interval number [22], evidence theory [17] and INS theory [9, 10], are proposed to process the imprecise evaluation data. However, these approaches supporting the compensation of multiple criteria neither identify the tolerable/unbearable threshold nor distinguish the minor/major difference for sensitive users. Thus, they cannot eliminate the possibility of selecting an invalid or unsatisfactory CS as the optimal one for SMEs.

As a type of non-compensatory method, ELECTRE III is more appropriate to handle the above problem. However, the traditional ELECTRE III method is difficult to apply directly to it due to some deficiencies [26], such as the limited type of data not supporting fuzzy evaluations, the complicated process of ranking computation, and the partial order of all alternatives. Especially, the traditional ELECTRE III method requires users to provide exact threshold values for ensuring the correct ranking result of candidates. Obviously, this is not practical.

Thus, this paper proposes an SMEs-oriented trustworthy CS recommendation approach based on non-compensatory MCDM. In it, a type of client context is viewed as an independent evaluation criterion, and the INNs are employed to measure the fuzzy trustworthiness of CSs. Based on the investigated outranking relations of INNs, the non-compensatory MCDM procedure via an improved ELECTRE III method with INS is developed to rank CSs.

4 Outranking relations of INNs for an improved ELECTRE III

The classic ELECTRE III method calculates the ranking values of all alternatives based on the single-valued evaluations. To overcome the limitations of single-valued data that fails depict the performance variation of CSs in an uncertain cloud environment, this paper defines the trustworthiness INN to describe the fuzzy trustworthiness of CSs. The outranking relations of INNs which will be employed in the improved ELECTRE III method are also established.

Definition 1. Trustworthiness INN [10]. A trustworthiness INN of CS A is characterized by T_A = <O_A, U_A, R_A >. O_A = [infO_A, supO_A] is the interval evaluation value of the CS’s performance, equivalent to the truth-membership function of the INN. R_A = [infR_A, supR_A] is the interval evaluation value of the potential risk, equivalent to the falsity-membership function of the INN. U_A = [infU_A, supU_A] is the interval evaluation value of the uncertainty of O_A and R_A, equivalent to the indeterminacy-membership function of the INN. O_A, U_A, R_A ∈ [0, 1], and 0 ≤ supO_A+supU_A+supR_A ≤ 3.

For example, a trustworthiness INN of CS A T_A = < [0.8, 0.9], [0.2, 0.3],[0.1, 0.3]>means that the performance of A is scored as [0.8, 0.9], the potential risk of A is scored as [0.2, 0.3], and the uncertainty of A is scored as [0.1, 0.3]. According to Definition 1, a trustworthy CS should possess a high-performance score and low potential risk and uncertainty scores.

Definition 2. Trustworthiness evaluation of a CS. Assume T_A = < [infO_A, supO_A], [infU_A, supU_A], [infR_A, supR_A] >and T_B = < [infO_B, supO_B], [infU_B, supU_B], [infR_B, supR_A] >are two trustworthiness INNs of CSs A and B, respectively. The trustworthiness evaluations of A and B is defined by $\begin{matrix} g (T_{A}) = (inf O_{A} + sup O_{A}) - (inf U_{A} + sup U_{A}) \\ - (inf R_{A} + sup R_{A}) \\ g (T_{B}) = (inf O_{B} + sup O_{B}) - (inf U_{B} + sup U_{B}) \\ - (inf R_{B} + sup R_{B}) \end{matrix}$ (1) The trustworthiness difference between CSs A and B is calculated as follows: $\begin{matrix} \begin{matrix} \frac{n!}{r! (n - r)!} G (T_{A}, T_{B}) = g (T_{A}) - g (T_{B}) \\ = g (O_{AB}) + g (U_{AB}) + g (R_{AB}) \end{matrix} \end{matrix},$ where g(O_AB) = (infO_A+supO_A)–(infO_B+supO_B), g(U_AB) = (infU_B +supU_B)–(infU_A+supU_A) and g(R_AB) = (infR_B+supR_B)–(infR_A+supR_A).

Definition 3. Let p and q (0 ≤ q ≤ p) represent the preference threshold and indifference threshold, respectively. The following relations between the INNs of CSs A and B are defined:

If p <G(T_A, T_B), then T_A is strictly preferred to T_B, denoted by T_A > _ST_B. Otherwise, T_A is not strictly preferred to T_B, denoted by T_A_{$>$
S}T_B.

If q <G(T_A, T_B) ≤ p, then T_A is weakly preferred to T_B, denoted by T_A > _WT_B. Otherwise, T_A is not weakly preferred to T_B, denoted by T_A_{$>$
W}T_B.

If –q ≤ G(T_A, T_B) ≤ q, then T_A is indifferent to T_B, denoted by T_A ∼ _IT_B.

If G(T_A, T_B) < –q, namely G(T_B, T_A) > q, then relation between T_A and T_B could be T_B > _ST_A or T_B > _WT_A according to (1) and (2).

Obviously, there are three binary relations between the INNs of CSs A and B. They are strong dominance relation, weak dominance relation, and indifference relation. If T_A is strictly preferred to T_B, it means that T_A strongly dominates T_B. If T_A is weakly preferred to T_B, it means that T_A weakly dominates T_B.

Example 2. Assume the preference threshold p = 0.4 and indifference threshold q = 0.2.

If T_A = < [0.7, 0.8],[0.0, 0.1],[0.1, 0.2]>and T_B = < [0.6, 0.7], [0.1, 0.2],[0.2, 0.3]>are two trustworthiness INNs of CSs A and B, respectively, then G(T_A, T_B) = 0.6 > p. Hence, the relation between T_A and T_B is that T_A > _ST_B, that is, T_A is strictly preferred to T_B.

If T_A = < [0.7, 0.9],[0.1, 0.2],[0.2, 0.3]>and T_B = < [0.6, 0.7], [0.1, 0.2],[0.2, 0.3]>are two trustworthiness INNs of CSs A and B, respectively, then q <G(T_A, T_B) = 0.3 ≤ p. Thus, the relation between T_A and T_B is that T_A > _WT_B, that is, T_A is weakly preferred to T_B.

If T_A = < [0.6, 0.7],[0.1, 0.2],[0.1, 0.3]>and T_B = < [0.6, 0.7], [0.1, 0.2],[0.2, 0.3]>are two trustworthiness INNs of CSs A and B, respectively, then –q <G(T_A, T_B) = 0.1 ≤ q. Therefore, T_A is indifferent to T_B, that is, T_A ∼ _IT_B.

Property 1. Let T_A, T_B and T_C be three trustworthiness INNs of CSs A, B and C, respectively. If T_A > _ST_B and T_B > _ST_C, then T_A > _ST_C.

Proof. According to Definition 3, if T_A > _ST_B then G(T_A, T_B) > p, that is, g(T_A)–g(T_B) > p. Similarly, if T_B > _ST_C g(T_B)–g(T_C) > p. We have g(T_A)–g(T_B)+g(T_B)–g(T_C) >2p. Thus, g(T_A)–g(T_C) > p, that is, T_A > _ST_C is achieved. The property is proven.

Definition 4. Let p and v (0 ≤ p ≤ v) represent the preference threshold and veto threshold, respectively. The following relations between two INNs are defined:

If v <G(T_A, T_B), then T_A is strongly opposed to T_B, denoted by T_A > _SOT_B. Otherwise, T_A is not strongly opposed to T_B, denoted by T_A_{$>$
SO}T_B.

If p <G(T_A, T_B) ≤ v, then T_A is weakly opposed to T_B, denoted by T_A > _WOT_B. Otherwise, T_A is not weakly opposed to T_B, denoted by T_A_{$>$
WO}T_B.

If –p <G(T_A, T_B) ≤ p, then T_A is indifferently opposed to T_B, denoted by T_A ∼ _IOT_B.

If G(T_A, T_B) < –p, namely G(T_B,T_A) > p, then the relation between T_A and T_B could be T_B > _SOT_A or T_B > _WOT_A according to (1) and (2).

Property 2. Let T_A, T_B and T_C be three trustworthiness INNs of CSs A, B and C, respectively, if T_A > _SOT_B and T_B > _SOT_C, then T_A > _SOT_C.

Proof. According to Definition 4, if T_A > _SOT_B then G(T_A, T_B) > v, that is, g(T_A)–g(T_B) > v. Similarly, if T_B > _SOT_C then g(T_B)–g(T_C) > v. Therefore, g(T_A)–g(T_B)+g(T_B)–g(T_C) >2v. Thus, g(T_A)–g(T_C) > v. Then T_A > _SOT_C is achieved. The property is proven.

Definition 5. Let $T_{A}^{C} = {T_{A, 1}, T_{A, 2}, . . ., T_{A, m}}$ and $T_{B}^{C} = {T_{B, 1}, T_{B, 2}, . . ., T_{B, m}}$ be the collections of trustworthiness INNs about CSs A and B, respectively. T_A_,j and T_B_,j denote the trustworthiness INNs of A and B, respectively, on criterion #j (j = 1, 2, ..., m). p_j and q_j are the preference threshold and indifference threshold, respectively, on criterion #j. To validate the hypothesis “A outranks B” for a given criterion, the concordance index between CSs A and B is defined as follows: $c_{j} (A, B) = {\begin{matrix} 0, if g_{j} (T_{B}^{C}) - g_{j} (T_{A}^{C}) \geq p_{j} \\ 1, if g_{j} (T_{B}^{C}) - g_{j} (T_{A}^{C}) \leq q_{j} \\ \frac{p_{j} - (g_{j} (T_{B}^{C}) - g_{j} (T_{A}^{C}))}{p_{j} - q_{j}}, otherwise \end{matrix},$ (2) where $g_{j} (T_{A}^{C})$ and $g_{j} (T_{B}^{C})$ are the trustworthiness evaluations of A and B, respectively, on criterion #j.

Similarly, let v_j and p_j represent the veto threshold and the preference threshold, respectively, on criterion #j. The discordance index for rejecting the assertion “A outranks B” on criterion #j (j = 1,2, ...,m) is defined as follows: $d_{j} (A, B) = {\begin{matrix} 0, if g_{j} (T_{B}^{C}) - g_{j} (T_{A}^{C}) \leq p_{j} \\ 1, if g_{j} (T_{B}^{C}) - g_{j} (T_{A}^{C}) \geq v_{j} \\ \frac{(g_{j} (T_{B}^{C}) - g_{j} (T_{A}^{C})) - p_{j}}{v_{j} - p_{j}}, otherwise \end{matrix}$ (3)

Definition 6. The index of global concordance is defined by

$\begin{matrix} C (A, B) = \frac{1}{W} \sum_{j = 1}^{m} w_{j} c_{j} (A, B), \\ W = \sum_{j = 1}^{m} w_{j}, j = 1 . . . m, \end{matrix}$ (4) where w_j represents the weight of criterion #j. C(A, B) tests the hypothesis that “A outranks B” among all the criteria.

The concordance index characterizes the strength of the positive arguments to validate the assertion that A is better than B. In general, if A is better than B, there may be some criteria where A is worse than B. And the discordance index is employed to verify the hypothesis “A is worse than B” for a given criterion. The veto threshold for each criterion is assigned to introduce discordance into the outranking relations.

Definition 7. The credibility index of outranking relation between CSs A and B is defined by $σ (A, B) = {\begin{matrix} C (A, B), if F = φ \\ C (A, B) \prod_{j \in F} \frac{1 - d_{j} (A, B)}{1 - C (A, B)}, if F \neg = φ \end{matrix},$ (5) where F is criteria set satisfying d_j (A, B) > C (A, B). If d_j (A, B) =1, σ (A, B) =0.

Definition 8. Assuming that {A₁, A₂, ..., A_n} are n candidate CSs, the dominance index of the pair (A_i, A_k) is defined by: $d σ (A_{i}, A_{k}) = σ (A_{i}, A_{k}) - σ (A_{k}, A_{i}) .$ (6)

The dominance index denotes the difference of the credibility index between each pair.

Definition 9. Assuming that {A₁, A₂, ..., A_n} are n candidate CSs, the net dominance index of each CS is defined as follows: $I (A_{i}) = {\sum_{j = 1}}_{j \neg = i}^{n} d σ (A_{i}, A_{j}) .$ (7)Definition 10. The most trustworthy CS A^* among {A₁, A₂, ..., A_n} satisfies: $I (A^{*}) = {max}_{i = 1}^{n} {I (A_{i})} .$ (8)

Then the sequence with the complete order of candidate CSs is established.

It should be noted that the traditional ELECTRE III method defines the ranking index to rank the alternatives. According to the ranking index, two complete preorders Z₁ and Z₂ will be constructed and the final result Z = Z₁ ∩ Z₂ is obtained. Frequently, however, the two orders are not the same and the final ranking is usually not a complete order. As a consequence, the approach in this paper adopts Equations (1)–(8) to construct the ranking and finally explores a complete order.

5 Non-compensatory MCDM procedure via an improved ELECTRE III

By applying the outranking relations of INNs, a non-compensatory MCDM procedure via an improved ELECTRE III is proposed to recommend the trustworthy CSs for SMEs, under the situation that assessment information on criteria is INNs.

5.1 Overall framework

The overall framework of the proposed MCDM procedure is shown in Fig. 2.

Fig. 2

The proposed non-compensatory MCDM procedure.

Assume that n candidate CSs {A₁, A₂, ..., A_n} could provide the function indeed matching with the requirements of an SME. Next, the client context of an SME can be analyzed. A type of client device in different ASs can be viewed as a type of client context, namely an independent evaluation criterion. The desktop computer, palmtop and cellular phone are the most common types of client devices [15]. Assuming that the SME consisting of multiple individual users is scattered in l ASs, the total number of decision criteria is m = 3×l. Assume that P = {p₁, p₂, ..., p_m}, Q = {q₁, q₂, ..., q_m}, and V = {v₁, v₂, ..., v_m} are the preference threshold, indifference threshold, and veto threshold for m decision criteria, respectively. W = {w₁, w₂, ..., w_m} represents the weights of decision criteria, 0 ≤ w_j ≤ 1, and $\sum_{j = 1}^{m} w_{j} = 1$ .

5.2 Pre-processing original data into trustworthiness INNs

The original data is pre-processed into the trustworthiness INNs as follows:

Step 1: Collect the evaluation data about the performance and potential risks of candidate CSs. Different types of CSs have different evaluation indicators of performance and potential risks [9]. Taking the performance for example, the original evaluation data of n candidate CSs related to the m types of client context is represented by: $O = {\begin{matrix} (o_{1, 1}^{1, 1}, o_{2, 1}^{1, 2}, \dots, o_{x_{1}, 1}^{1, H}) (o_{1, 2}^{1, 1}, o_{2, 2}^{1, 2}, \dots, o_{x_{2}, 2}^{1, H}) . . . . . . (o_{1, m}^{1, 1}, o_{1, m}^{1, 2}, \dots, o_{x_{m}, m}^{1, H}) \\ (o_{1, 1}^{2, 1}, o_{2, 1}^{2, 2}, \dots, o_{x_{1}, 1}^{2, H}) (o_{1, 2}^{2, 1}, o_{2, 2}^{2, 2}, \dots, o_{x_{2}, 2}^{2, H}) . . . . . . (o_{1, m}^{2, 1}, o_{1, m}^{2, 2}, \dots, o_{x_{m}, m}^{2, H}) \\ . . . . . . . . . . . . . . . . . . . . . . . . \\ (o_{1, 1}^{n, 1}, o_{2, 1}^{n, 2}, \dots, o_{x_{1}, 1}^{n, H}) (o_{1, 2}^{n, 1}, o_{2, 2}^{n, 2}, \dots, o_{x_{2}, 2}^{n, H}) . . . . . . (o_{1, m}^{n, 1}, o_{1, m}^{n, 2}, \dots, o_{x_{m}, m}^{n, H}) \end{matrix}},$ (9) where $o_{i, j}^{a, k}$ represents the evaluation value of the kth performance indicator of CS a from user #i in the jth client context; H is the number of performance indicators; x_j is the total number of users in the jth client context. The multiple-attribute evaluation of potential risks can also be defined similarly by Equation (9).

Step 2: Aggregate the multi-dimensional performance evaluations and risk evaluations into the comprehensive evaluations with the weighted arithmetic averaging operator. The gain-type performance indicators can be normalized by: $o_{i, j}^{a, k} = (o_{i, j}^{a, k} - {min}_{a = 1}^{n} o_{i, j}^{a, k}) / ({max}_{a = 1}^{n} o_{i, j}^{a, k} - {min}_{a = 1}^{n} o_{i, j}^{a, k}) .$ (10)

The loss-type performance indicators can be normalized by: $o_{i, j}^{a, k} = ({max}_{a = 1}^{n} o_{i, j}^{a, k} - o_{i, j}^{a, k}) / ({max}_{a = 1}^{n} o_{i, j}^{a, k} - {min}_{a = 1}^{n} o_{i, j}^{a, k}) .$ (11)

Let $W^{I} = {w_{1}^{I}, w_{2}^{I}, \dots, w_{H}^{I}}$ be the weights vector of performance indicators. $w_{k}^{I}$ represents the weight of the kth indicator. $0 \leq w_{k}^{I} \leq 1$ and $\sum_{k = 1}^{H} w_{k}^{I} = 1$ . The comprehensive performance evaluation of CS a from user #i in the jth client context can be determined by: $e_{i, j}^{O} (a) = \sum_{k = 1}^{H} (o_{i, j}^{a, k} \times w_{k}^{I}) .$ (12)

Similarly, the comprehensive potential risk evaluation from user #i in the jth client context, $e_{i, j}^{R} (a)$ , can also be obtained.

Step 3: Transform the single-value evaluation data from the same client context into the interval numbers by utilizing the cloud model theory. The cloud model [46] is a cognitive model realizing the bidirectional transformation between a qualitative concept and the quantitative data based on the probability statistics and the fuzzy set theory. It can effectively represent the fuzziness, randomness and uncertain concepts, and has been applied in many fields [46 –48]. In this paper, we establish the cloud models for performance and potential risks to identify their interval numbers in different client context. Let $E_{j}^{O} (a) = {e_{1, j}^{O} (a), e_{2, j}^{O} (a), . . . ., e_{x_{j}, j}^{O} (a)}$ be the performance data of CS a from x_j users in the jth client context. The data is viewed as cloud drops and sent into the reverse cloud generator. The performance cloud model of CS a in the jth client context is obtained as follows: ${\begin{matrix} {Ex}_{a, j}^{O} = \frac{1}{x_{j}} \times \sum_{k = 1}^{x_{j}} e_{k, j}^{O} (a) \\ {En}_{a, j}^{O} = \sqrt{\frac{π}{2}} \times \frac{1}{x_{j}} \sum_{k = 1}^{x_{j}} | e_{k, j}^{O} (a) - {Ex}_{a, j}^{O} | \\ {HE}_{a, j}^{O} = \sqrt{| \frac{1}{x_{j} - 1} \sum_{k = 1}^{x_{j}} {(e_{k, j}^{O} (a) - {Ex}_{a, j}^{O})}^{2} - {({En}_{a, j}^{O})}^{2} |} \end{matrix} .$ (13)

The performance of CS a in the jth client context is defined as o_a,j, described with an interval number $O_{a, j} = [o_{a, j}^{L}, o_{a, j}^{U}]$ . $o_{a, j}^{L}$ and $o_{a, j}^{U}$ are calculated by: ${\begin{matrix} o_{a, j}^{U} = {Ex}_{a, j}^{O} + {En}_{a, j}^{O} + {HE}_{a, j}^{O} \times τ \\ o_{a, j}^{L} = {Ex}_{a, j}^{O} - {En}_{a, j}^{O} - {HE}_{a, j}^{O} \times τ \end{matrix},$ (14) where τ is the influence coefficient of HE, which is recommended to remain in the interval range [0.1,0.2] [22]. Similarly, the cloud model for potential risks can also be established, and the potential risks interval number of CS a in the jth client context is defined as $R_{a, j} = [r_{a, j}^{L}, r_{a, j}^{U}]$ .

Step 4: Calculate the uncertainty interval of CS a in the jth client context. Let $λ_{a, j}^{O} = \sqrt{{({En}_{a, j}^{O})}^{2} + {({HE}_{a, j}^{O})}^{2}}$ and $λ_{a, j}^{R} = \sqrt{{({En}_{a, j}^{R})}^{2} + {({HE}_{a, j}^{R})}^{2}}$ be the uncertainty of performance and potential risks of a in the jth client context, respectively. The uncertainty interval number of a in the jth client context is defined by:

$\begin{matrix} U_{a, j} & = [u_{a, j}^{L}, u_{a, j}^{U}] \\ = [min {λ_{a, j}^{O}, λ_{a, j}^{R}}, max {λ_{a, j}^{O}, λ_{a, j}^{R}}] . \end{matrix}$ (15)

Step 5: Assemble all the pre-processed data into the trustworthiness INNs matrix. The original data of n candidate CSs in m types of client context is transformed into the trustworthiness INNs matrix by $\begin{matrix} T^{C} = (\begin{matrix} T_{1} \\ T_{2} \\ ⋮ \\ T_{n} \end{matrix}) = (\begin{matrix} T_{1, 1} T_{1, 2} \dots T_{1, m} \\ T_{2, 1} T_{2, 2} \dots T_{2, m} \\ ⋮ ⋮ ⋮ ⋮ \\ T_{n, 1} T_{n, 2} \dots T_{n, m} \end{matrix}) = \\ (\begin{matrix} 〈 O_{1, 1}, U_{1, 1}, R_{1, 1} 〉 & \dots & 〈 O_{1, m}, U_{1, m}, R_{1, m} 〉 \\ ⋮ & ⋱ & ⋮ \\ 〈 O_{n, 1}, U_{n, 1}, R_{n, 1} 〉 & \dots & 〈 O_{n, m}, U_{n, m}, R_{n, m} 〉 \end{matrix}), \end{matrix}$ where $T_{i, j} = 〈 O_{i, j}, U_{i, j}, R_{i, j} 〉 = 〈 [o_{i, j}^{L}, o_{i, j}^{U}], [u_{i, j}^{L}, u_{i, j}^{U}], [r_{i, j}^{L}, r_{i, j}^{U}] 〉$ is the trustworthiness INN of CS #i in the jth client context.

5.3 Normalization of threshold values

In the traditional ELECTRE III method, the preference threshold (P), indifference threshold (Q), and veto threshold (V) are defined by absolute values. However, the performance of CSs is dynamic and uncertain due to the susceptibility and vulnerability of cloud environment. The performance difference between two candidate CSs may be very small in one application scenario, but could also be significantly large in another scenario. Users can hardly predict the appropriate values for P, Q, and V. Thus, the exact thresholds are not suitable for the SMEs-oriented CS recommendation problem.

In this paper, we propose an innovation that employs the relative ratio rather than absolute values to define the thresholds for improving the ELECTRE III method. The performance evaluation values of different CSs may be quite different in both the traditional ELECTRE III and our improved ELECTRE III with INS. According to Equation (1), the value of g(T_A) ranges from –4 to 2. When the performance difference between CSs is too large or too small, P, Q, and V cannot work effectively to solve the MCDM problem. Thus, the performance evaluation values of candidate CSs must be normalized. Assuming that g_j (T_{A
_i}) (i = 1,2,...,n; j = 1,2, ..., m) represents the performance evaluation of CS A_i on criteria #j, its normalized value is defined by $g_{j}^{N} (T_{A_{i}}) = \frac{g_{j} (T_{A_{i}}) - {min}_{i = 1}^{n} g_{j} (T_{A_{i}})}{{max}_{i = 1}^{n} g_{j} (T_{A_{i}}) - {min}_{i = 1}^{n} g_{j} (T_{A_{i}})} .$ (16)

The performance difference between CSs A_i and A_k, denoted by G_j (T_{A
_i}, T_{A
_k}), also becomes a normalized value within [0, 1]. Next, we limit the range of P, Q, and V to [0, 1] and the thresholds related to criterion #j satisfy 0 ≤ q_j <p_j <v_j ≤ 1. Then, the preference threshold, indifference threshold, and veto threshold become the relative ratio values. In this case, the suggested values of P, Q and V may be given for different types of users as follows:

For the cautious SMEs which are very sensitive to the CSs’ QoS, the range of Q is [0, 0.10], the range of P is [0.15, 0.20], and the range of V is [0.25, 0.30].

For the moderate SMEs which could accept the variation of the CSs’ QoS to some extent, the range of Q is [0.10, 0.15], the range of P is [0.20, 0.25], and the range of V is [0.30, 0.40].

For the optimistic SMEs which could tolerate a wide variation range of the CSs’ QoS, the range of Q is [0.10, 0.20], the range of P is [0.25, 0.35], and the range of V is [0.40, 0.50].

5.4 Solving steps

Based on the above preparations, a procedure to rank candidate CSs is given as follows:

Step 1: Compute the trustworthiness difference matrix of candidate CSs. The normalized trustworthiness evaluation matrix of n CSs {A₁, A₂, ..., A_n} on m criteria, namely m types of client context, is denoted as follows: $G^{N} = [\begin{matrix} g_{1}^{N} (T_{A_{1}}) & g_{2}^{N} (T_{A_{1}}) & . . . & g_{m}^{N} (T_{A_{1}}) \\ g_{1}^{N} (T_{A_{2}}) & g_{2}^{N} (T_{A_{2}}) & . . . & g_{m}^{N} (T_{A_{2}}) \\ . . . & . . . & . . . & . . . \\ g_{1}^{N} (T_{A_{n}}) & g_{2}^{N} (T_{A_{n}}) & . . . & g_{m}^{N} (T_{A_{n}}) \end{matrix}],$ where $g_{j}^{N} (T_{A_{i}})$ represents the normalized trustworthiness of A_i on criterion #j obtained by Equation (16).

Then, calculate the trustworthiness difference G_j (T_{A
_i}, T_{A
_k}) between any two CSs on each criterion. $G_{j} (T_{A_{i}}, T_{A_{k}}) = g_{j}^{N} (T_{A_{i}}) - g_{j}^{N} (T_{A_{k}})$ (i,k = 1,2,...,n; j = 1,2, ..., m) represents the trustworthiness difference between CSs A_i and A_k on criterion #j. A trustworthiness difference matrix is created as follows: $D = [\begin{matrix} G_{1} (T_{A_{1}}, T_{A_{1}}) & G_{2} (T_{A_{1}}, T_{A_{1}}) & . . . & G_{m} (T_{A_{1}}, T_{A_{1}}) \\ G_{1} (T_{A_{1}}, T_{A_{2}}) & G_{2} (T_{A_{1}}, T_{A_{2}}) & . . . & G_{m} (T_{A_{1}}, T_{A_{2}}) \\ . . . & . . . & . . . & . . . \\ G_{1} (T_{A_{n}}, T_{A_{n}}) & G_{2} (T_{A_{n}}, T_{A_{n}}) & . . . & G_{m} (T_{A_{n}}, T_{A_{n}}) \end{matrix}] .$

Step 2: Calculate the concordance index c_j(A_i, A_k) between each pair (A_i, A_k) on each criterion using Equation (2). The concordance index matrix is constructed by $c^{M} = [\begin{matrix} c_{1} (A_{1}, A_{1}) & c_{2} (A_{1}, A_{1}) & . . . & c_{m} (A_{1}, A_{1}) \\ c_{1} (A_{1}, A_{2}) & c_{2} (A_{1}, A_{2}) & . . . & c_{m} (A_{1}, A_{2}) \\ . . . & . . . & . . . & . . . \\ c_{1} (A_{n}, A_{n}) & c_{2} (A_{n}, A_{n}) & . . . & c_{m} (A_{n}, A_{n}) \end{matrix}] .$

Step 3: Calculate the global concordance index between each pair (A_i, A_k) on all criteria. By Equation (4), the global concordance index (A_i, A_k) between each pair (A_i, A_k) on all criteria is computed as follows: $C (A_{i}, A_{k}) = \frac{1}{W} \sum_{j = 1}^{m} w_{j} c_{j} (A_{i}, A_{k}) .$

Thus, the global concordance matrix C with all candidate CSs is denoted as follows: $C = [\begin{matrix} 1 & c_{12} & c_{13} & . . . & c_{1 (n - 1)} & c_{1 n} \\ c_{21} & 1 & c_{23} & . . . & c_{2 (n - 1)} & c_{2 n} \\ . . . & . . . & 1 & . . . & . . . & . . . \\ c_{(n - 1) 1} & c_{(n - 1) 2} & c_{(n - 1) 3} & . . . & 1 & c_{(n - 1) n} \\ c_{n 1} & c_{n 2} & c_{n 3} & . . . & c_{n (n - 1)} & 1 \end{matrix}],$ where c_ik refers to C(A_i, A_k) (i,k = 1,2,...,n).

Step 4: Calculate the discordance index of each pair (A_i, A_k) on every criterion. The discordance index d_j(A_i, A_k) of each pair (A_i, A_k) on criterion #j is computed by Equation (3). The discordance index matrix can be constructed as follows: $d^{M} = [\begin{matrix} d_{1} (A_{1}, A_{1}) & d_{2} (A_{1}, A_{1}) & . . . & d_{m} (A_{1}, A_{1}) \\ d_{1} (A_{1}, A_{2}) & d_{2} (A_{1}, A_{2}) & . . . & d_{m} (A_{1}, A_{2}) \\ . . . & . . . & . . . & . . . \\ d_{1} (A_{n}, A_{n}) & d_{2} (A_{n}, A_{n}) & . . . & d_{m} (A_{n}, A_{n}) \end{matrix}] .$

Step 5: Compute the credibility index of each pair (A_i, A_k) by Equation (5). The credibility index matrix can be constructed by: $σ^{M} = [\begin{matrix} σ (A_{1}, A_{1}) & σ (A_{1}, A_{2}) & . . . & σ (A_{1}, A_{n}) \\ σ (A_{2}, A_{1}) & σ (A_{2}, A_{2}) & . . . & σ (A_{2}, A_{n}) \\ . . . & . . . & . . . & . . . \\ σ (A_{n}, A_{1}) & σ (A_{n}, A_{2}) & . . . & σ (A_{n}, A_{n}) \end{matrix}] .$

Table 2
Trustworthiness INNS of 3 CSs

Criterion #1 Criterion #2 Criterion #3 Criterion #4 Criterion #5 Criterion #6

O U R O U R O U R O U R O U R O U R

A ₁ 1.46 1.46 0 0 0 0 1.64 1.64 0 0 0 0 1.38 1.38 0 0 0 0 1.38 1.38 0 0 0 0 1.46 1.46 0 0 0 0 2.14 2.14 0 0 0 0

A ₂ 1.15 1.15 0 0 0 0 1.20 1.20 0 0 0 0 1.18 1.18 0 0 0 0 1.14 1.14 0 0 0 0 1.2 1.2 0 0 0 0 3.50 3.50 0 0 0 0

A ₃ 1.62 1.62 0 0 0 0 1.82 1.82 0 0 0 0 1.52 1.52 0 0 0 0 1.51 1.51 0 0 0 0 1.6 1.6 0 0 0 0 1.41 1.41 0 0 0 0

	Criterion #1	Criterion #2
A ₁	1.46	1.46	1.64	1.64	1.38	1.38	1.38	1.38	1.46	1.46	2.14	2.14
A ₂	1.15	1.15	1.20	1.20	1.18	1.18	1.14	1.14	1.2	1.2	3.50	3.50
A ₃	1.62	1.62	1.82	1.82	1.52	1.52	1.51	1.51	1.6	1.6	1.41	1.41

Step 6: Calculate the dominance index of the pair (A_i, A_k) according to Equation (6). The dominance index matrix can be constructed as follows: $d σ^{M} = [\begin{matrix} d σ (A_{1}, A_{1}) & d σ (A_{1}, A_{2}) & . . . & d σ (A_{1}, A_{n}) \\ d σ (A_{2}, A_{1}) & d σ (A_{2}, A_{2}) & . . . & d σ (A_{2}, A_{n}) \\ . . . & . . . & . . . & . . . \\ d σ (A_{n}, A_{1}) & d σ (A_{n}, A_{2}) & . . . & d σ (A_{n}, A_{n}) \end{matrix}] .$

Step 7: Compute the net dominance index of one CS with each other based on Equation (7). For each CS, the net dominance index can be obtained as follows: $I (A_{i}) = {\sum_{k = 1}}_{k \neg = i}^{n} d σ (A_{i}, A_{k}) .$

Step 8: Rank CSs according to I(A_i). The complete order for candidate CSs is explored, and the most trustworthy CS A^* will be recommended to the SME.

6 Experiments

To illustrate the effectiveness of the proposed approach, the following experiments are executed.

6.1 Case analysis based on simulation data

Firstly, we apply the improved ELECTRE III method with INS to Example 1.

The weights of criteria #1-#6 are set as: W = {1/6, 1/6, 1/6, 1/6, 1/6, 1/6}. Assuming the SME is a moderate user, the thresholds of every criterion are given as: P = {0.20, 0.20, 0.20, 0.20, 0.20, 0.20}, Q = {0.10, 0.10, 0.10, 0.10, 0.10, 0.10} and V = {0.30, 0.30, 0.30, 0.30, 0.30, 0.30}. The response time data in Table 1 are used to generate the trustworthiness INNs of CSs. In Table 1, the performance of each CS is given by a single value. Thus, we employ a specific interval number, which has the same upper limit and lower limit, to describe the performance of a CS. Moreover, considering that the evaluations of potential risk are missing, we employ [0, 0] to describe the interval evaluation of the potential risk or uncertainty. Based on Equations (14) and (15), the trustworthiness INNs of 3 CSs (i.e., A₁-A₃) are obtained as shown in Table 2.

Next, the MCDM procedure via the improved ELECTRE III with INS is as follows:

Firstly, calculate the performance matrix. The normalized trustworthiness is as follows: $G^{N} = [\begin{matrix} 0.0000 & 0 . 0000 & 0 . 0000 & 0 . 0000 & 0 . 0000 & 0 . 6507 \\ 0 . 5155 & 0 . 4639 & 0 . 5308 & 0 . 5598 & 0 . 5200 & 0 . 0000 \\ 1 . 0000 & 1 . 0000 & 1 . 0000 & 1 . 0000 & 1 . 0000 & 1 . 0000 \end{matrix}] .$

Obtain the discordance index matrix, and then the credibility index matrix between candidates is as follows: $σ^{M} = [\begin{matrix} 1 . 00 00 & 0 . 00 00 & 0 . 00 00 \\ 0 . 00 00 & 1 . 00 00 & 0 . 00 00 \\ 1 . 00 00 & 1 . 00 00 & 1 . 00 00 \end{matrix}] .$

The dominance index matrix is as follows: $d σ^{M} = [\begin{matrix} 0 . 00 00 & 0 . 00 00 & - 1 . 00 00 \\ 0 . 00 00 & 0 . 00 00 & - 1 . 00 00 \\ 1 . 00 00 & 1 . 00 00 & 0 . 00 00 \end{matrix}] .$

The net dominance matrix is I = [–1.0000, –1.0000, 2.0000]. Thus, the ranking list of CSs is: A₃≻A₂≻A₁. Obviously, the improved ELECTRE III method via INS can remove all distractions from the invalid candidate A₂ or unsatisfactory candidate A₁ and recommend the optimal A₃ to an SME.

6.2 Case analysis based on real dataset

To demonstrate our approach, we also use WS-DREAM dataset #1 [49], which describes real-world QoS evaluation results obtained from 339 users on 5,825 services. The dataset has been applied to research concerned with cloud computing [17 , 50]. Considering that no client devices information is given in this dataset, the AS information is employed to differentiate the client contexts. The statistics show that 339 users are distributed in 138 ASs. 4 ASs that contain more than 6 users are listed in Table 3.

Table 3
Trustworthiness INNs of candidate CSs

Criterion #1 Criterion #2 Criterion #3 Criterion #4

O U R O U R O U R O U R

A ₁ 0.00 0.62 0.16 1.00 0.02 0.31 0.00 0.89 0.61 1.00 0.09 1.00 0.00 1.00 0.37 1.00 0.00 0.60 0.31 0.71 0.07 0.81 0.00 0.10

A ₂ 0.97 0.99 0.00 0.08 0.00 0.00 0.99 1.00 0.00 0.15 0.00 0.00 0.84 0.99 0.00 0.57 0.00 0.00 0.92 0.93 0.00 0.03 0.00 0.00

A ₃ 0.52 0.98 0.13 1.00 0.00 0.13 0.96 0.98 0.00 0.20 0.00 0.00 0.91 0.96 0.00 0.17 0.00 0.00 0.76 0.81 0.00 0.13 0.00 0.00

A ₄ 0.44 0.92 0.14 1.00 0.00 0.14 0.93 0.97 0.00 0.42 0.00 0.00 0.92 0.93 0.00 0.03 0.00 0.00 0.00 1.00 0.25 1.00 0.00 0.32

A ₅ 0.49 0.96 0.14 1.00 0.00 0.14 0.93 0.98 0.01 0.49 0.00 0.01 0.56 1.00 0.11 1.00 0.00 0.16 0.62 0.76 0.00 0.29 0.00 0.00

A ₆ 0.54 1.00 0.13 1.00 0.00 0.13 0.97 0.99 0.00 0.21 0.00 0.00 0.88 0.96 0.00 0.32 0.00 0.00 0.76 0.84 0.00 0.16 0.00 0.00

	Criterion #1	Criterion #2	Criterion #3	Criterion #4
A ₁	0.00	0.62	0.16	1.00	0.02	0.31	0.00	0.89	0.61	1.00	0.09	1.00	0.00	1.00	0.37	1.00	0.60	0.31	0.71	0.07	0.81	0.10
A ₂	0.97	0.99	0.00	0.08	0.00	0.00	0.99	1.00	0.00	0.15	0.00	0.00	0.84	0.99	0.00	0.57	0.00	0.92	0.93	0.00	0.03	0.00
A ₃	0.52	0.98	0.13	1.00	0.00	0.13	0.96	0.98	0.00	0.20	0.00	0.00	0.91	0.96	0.00	0.17	0.00	0.76	0.81	0.00	0.13	0.00
A ₄	0.44	0.92	0.14	1.00	0.00	0.14	0.93	0.97	0.00	0.42	0.00	0.00	0.92	0.93	0.00	0.03	0.00	0.00	1.00	0.25	1.00	0.32
A ₅	0.49	0.96	0.14	1.00	0.00	0.14	0.93	0.98	0.01	0.49	0.00	0.01	0.56	1.00	0.11	1.00	0.16	0.62	0.76	0.00	0.29	0.00
A ₆	0.54	1.00	0.13	1.00	0.00	0.13	0.97	0.99	0.00	0.21	0.00	0.00	0.88	0.96	0.00	0.32	0.00	0.76	0.84	0.00	0.16	0.00

Assuming that the individual users of a group are distributed in the above 4 ASs, services #35-#40 are selected as the candidate CSs, and their response time data is collected from the users in the 4 ASs. Services #35-#40 are denoted by A₁-A₆, respectively, and the 4 ASs, including AS680, AS559, AS2500 and AS786, are denoted by criteria #1-#4, respectively. The total number of users in each AS determines the weight of the criterion. The original data used in the experiments is given in Appendix A.1. Based on the data, the potential risks of services are evaluated by employing the same method in Ref. [9, 10].

According to the data, only 27 users from AS680 invoke CSs (i.e., A₁-A₆), and the evaluations of user #329 are missing. Then, the weights of criteria #1-#4 are set as: W = {27/52, 9/52, 9/52, 7/52}. The thresholds of each criterion are given as: P = {0.20, 0.20, 0.20, 0.20}, Q = 0.10, 0.10, 0.10, 0.10 and V = {0.30, 0.30, 0.30, 0.30}. Based on Equations (9)–(13), the original evaluation data is pre-processed into the performance cloud models and potential risk cloud models, and the results are given in Appendix A.2. Based on Equations (14) and (15), the trustworthiness INNs of candidate CSs are obtained as shown in Table 4.

Table 4

ASs used in experiments

AS number	Number of users	IDs of users
AS680	28	78,79,83,118,119,132,133,148,149,150,
		151,152,153,167,168,174,175,176,177,235,
		236,261,262,263,284,285,328,329
AS559	9	87,88,237,238,239,240,241,242,243
AS2500	9	212,213,214,215,303,304,305,306,307
AS786	7	29,114,115,144,145,246,269

Next, the MCDM procedure via the improved ELECTRE III is executed as follows:

Step 1: Calculate the performance matrix. The normalized trustworthiness is as follows: $G^{N} = [\begin{matrix} 0.0000 & 0.0000 & 0.0000 & 0.2556 \\ 1.0000 & 1.0000 & 0.8011 & 1.0000 \\ 0.4015 & 0.9748 & 0.9552 & 0.8427 \\ 0.3454 & 0.9021 & 1.0000 & 0.0000 \\ 0.3780 & 0.8796 & 0.4514 & 0.6966 \\ 0.4179 & 0.9737 & 0.8899 & 0.8417 \end{matrix}] .$

Obtain the trustworthiness difference matrix D as shown in Fig. 3(a).

Fig. 3

Values of some matrices. (a) D; (b) c^M; (c) d^M.

Step 2: Calculate the concordance index matrix c^M as shown in Fig. 3(b).

Step 3: Compute the global concordance index, and the result is shown as follows: $C = [\begin{matrix} 1.0000 & 0.0000 & 0.0000 & 0.1346 & 0.0000 & 0.0000 \\ 1.0000 & 1.0000 & 0.9064 & 0.8289 & 1.0000 & 1.0000 \\ 1.0000 & 0.4036 & 1.0000 & 1.0000 & 1.0000 & 1.0000 \\ 0.8654 & 0.3462 & 0.8654 & 1.0000 & 0.8654 & 0.8654 \\ 1.0000 & 0.1378 & 0.7649 & 0.8269 & 1.0000 & 0.7661 \\ 1.0000 & 0.4023 & 1.0000 & 0.9826 & 1.0000 & 1.0000 \end{matrix}] .$

Step 4: Obtain the discordance index matrix d^M as shown in Fig. 3(c).

Step 5: The credibility index matrix is as follows: $σ^{M} = [\begin{matrix} 1.0000 & 0.0000 & 0.0000 & 0.0000 & 0.0000 & 0.0000 \\ 1.0000 & 1.0000 & 0.9064 & 0.8289 & 1.0000 & 1.0000 \\ 1.0000 & 0.0000 & 1.0000 & 1.0000 & 1.0000 & 1.0000 \\ 0.8654 & 0.0000 & 0.0000 & 1.0000 & 0.0000 & 0.0000 \\ 1.0000 & 0.0000 & 0.0000 & 0.0000 & 1.0000 & 0.0000 \\ 1.0000 & 0.0000 & 1.0000 & 0.9826 & 1.0000 & 1.0000 \end{matrix}] .$

Step 6: The dominance index matrix between candidates is $d σ^{M} = [\begin{matrix} 0 . 0000 - 1 . 0000 - 1 . 0000 - 0 . 8654 - 1 . 0000 - 1 . 0000 \\ 1 . 0000 0 . 0000 0 . 9064 0 . 8289 1 . 0000 1 . 0000 \\ 1 . 0000 - 0 . 9064 0 . 0000 1 . 0000 1 . 0000 0 . 0000 \\ 0 . 8654 - 0 . 8289 - 1 . 0000 0 . 0000 0 . 0000 - 0 . 9826 \\ 1 . 0000 - 1 . 0000 - 1 . 0000 0 . 0000 0 . 0000 - 1 . 0000 \\ 1 . 0000 - 1 . 0000 0 . 0000 0 . 9826 1 . 0000 0 . 0000 \end{matrix}]$

Step 7: The final ranking, A₂ ≻ A₃ ≻ A₆ ≻ A₄ ≻ A₅ ≻ A₁, is obtained. The result shows that A₂ is the optimal CS for the group user. By reference to the original data, A₂ is indeed the candidate with the best performance and the lowest potential risks among all CSs.

6.3 Comparative analysis based on real dataset

A comparative study was done in order to verify the feasibility of the proposed approach. The proposed approach is compared with two other methods. The methods are described as follows: (1) Approach #1: the ELECTRE III with INS by employing the absolute values of thresholds to guide the decision;

(2) Approach #2: the traditional ELECTRE III with single values, in which the performance of a CS in a criterion is described with the average value of evaluations from all users in one AS, by employing the relative ratio values of thresholds to guide the decision; (3) Approach #3: the proposed approach, the ELECTRE III with INS by employing the relative ratio values of thresholds to guide the decision. In the experiments, we also investigate the influence of preference threshold (P), indifference threshold (Q), and veto threshold (V) on the ranking order of the candidates. Three types of users are simulated by setting different P, Q and V. For the cautious SMEs, P = {0.2, 0.2, 0.2, 0.2}, Q = {0.1, 0.1, 0.1, 0.1}, V = {0.3, 0.3, 0.3, 0.3}; For the moderate SMEs, P = {0.25, 0.25, 0.25, 0.25}, Q = {0.1, 0.1, 0.1, 0.1}, V = {0.4, 0.4, 0.4, 0.4}; For the optimistic SMEs, P = {0.3, 0.3, 0.3, 0.3}, Q = {0.1, 0.1, 0.1, 0.1}, V = {0.5, 0.5, 0.5, 0.5}. When the other setup is identical to the previous experiment, the comparison results are shown in Table 5. From Table 5, the analysis is given as follows:

Table 5
The result obtained by different approaches

Approach User type Net dominance index Ranking result

#1 cautious I = {–4.0000,2.8269,3.0991,–0.9565,–2.0000,1.0305} A₃≻A₂≻A₆≻A₄≻A₅≻A₁

moderate I = {–4.0000,2.8291,3.0661,–1.7979,–2.0000,1.9028} A₃≻A₂≻A₆≻A₄≻A₅≻A₁

optimistic I = {–4.0000,3.6987,2.2226,–1.8052,–2.0000,1.8838} A ₂ ≻ A ₃ ≻ A ₆ ≻ A₄ ≻ A₅ ≻ A₁

#2 cautious I = {–5.0000,5.0000,2.0000,–2.0000,–2.0000,2.0000} A₂ ≻ {A₃,A₆} ≻ (A₄,A₅) ≻ A₁

moderate I = {–5.0000,4.5539,1.6745,–2.8269,–0.0139,1.6123} A ₂ ≻ A ₃ ≻ A ₆ ≻ A₅ ≻ A₄ ≻ A₁

optimistic I = {–5.0000,4.2994,1.6033,–1.9615,–0.5793,1.6380} A₂≻A₆≻A₃≻A₅≻A₄≻A₁

#3 cautious I = {–4.8654,4.7353,2.0936,–1.9461,–2.0000,1.9826} A ₂ ≻ A ₃ ≻ A ₆ ≻ A₄ ≻ A₅ ≻ A₁

moderate I = {–4.8654,4.8235,2.0624,–2.0089,–2.0000,1.9884} A ₂ ≻ A ₃ ≻ A ₆ ≻ A₅ ≻ A₄ ≻ A₁

optimistic I = {–4.8952,4.8676,2.0468,–2.0105,–1.2035,1.1948} A ₂ ≻ A ₃ ≻ A ₆ ≻ A₅ ≻ A₄ ≻ A₁

Approach	User type	Net dominance index	Ranking result
#1	cautious	I = {–4.0000,2.8269,3.0991,–0.9565,–2.0000,1.0305}	A₃≻A₂≻A₆≻A₄≻A₅≻A₁
	moderate	I = {–4.0000,2.8291,3.0661,–1.7979,–2.0000,1.9028}	A₃≻A₂≻A₆≻A₄≻A₅≻A₁
	optimistic	I = {–4.0000,3.6987,2.2226,–1.8052,–2.0000,1.8838}	A ₂ ≻ A ₃ ≻ A ₆ ≻ A₄ ≻ A₅ ≻ A₁
#2	cautious	I = {–5.0000,5.0000,2.0000,–2.0000,–2.0000,2.0000}	A₂ ≻ {A₃,A₆} ≻ (A₄,A₅) ≻ A₁
	moderate	I = {–5.0000,4.5539,1.6745,–2.8269,–0.0139,1.6123}	A ₂ ≻ A ₃ ≻ A ₆ ≻ A₅ ≻ A₄ ≻ A₁
	optimistic	I = {–5.0000,4.2994,1.6033,–1.9615,–0.5793,1.6380}	A₂≻A₆≻A₃≻A₅≻A₄≻A₁
#3	cautious	I = {–4.8654,4.7353,2.0936,–1.9461,–2.0000,1.9826}	A ₂ ≻ A ₃ ≻ A ₆ ≻ A₄ ≻ A₅ ≻ A₁
	moderate	I = {–4.8654,4.8235,2.0624,–2.0089,–2.0000,1.9884}	A ₂ ≻ A ₃ ≻ A ₆ ≻ A₅ ≻ A₄ ≻ A₁
	optimistic	I = {–4.8952,4.8676,2.0468,–2.0105,–1.2035,1.1948}	A ₂ ≻ A ₃ ≻ A ₆ ≻ A₅ ≻ A₄ ≻ A₁

Approach #1 obtains the correct ranking result only when P, Q and V are set large enough. For the cautious and moderate SMEs, approach #1 identifies A₃ as the optimal one. Obviously, the performance of A₃ worse than A₂ according to Table 4. This shows the deficiency of approach #1. It is difficult to exactly predict the absolute values of thresholds in a specified application scenario.

Approach #2 finds the optimal CS by employing the relative ratio values of thresholds. However, the ranking results change when the thresholds are set to different values. The reason is that the static average value is difficult to exactly reflect the real performance of a CS, and the single evaluation value easily covers up the flaws of a CS if its performance is unreliable and unstable. In this case, it is completely possible that an invalid candidate is selected as the optimal CS. Fox example, assuming that {0.50, 5.40, 5.00, 5.00, 1.00, 0.50, 0.40, 0.20, 0.30, 0.40} and {1.97, 1.89, 1.85, 1.86, 1.79, 1.79, 1.92, 1.88, 1.95, 1.90} are the response time data from the users in one AS about CSs a and b, respectively, the average value of CS a, namely 1.87, is less than that of CS b, namely 1.88. Nevertheless, CS b obviously outperforms CS a because 30 percent of users experienced the long wait time of CS a and it is far beyond the expected 2 seconds.

Approach #3 analyzes the trustworthiness of CSs from three aspects including the performance, uncertainty and potential risk via the cloud model method and the INS theory. The data from each user may contribute to the objective and comprehensive evaluations of a CS, which facilitates to make the right decision. Table 5 shows that the ranking values of the top 3 CSs obtained by approach #3 are correct when the thresholds are set with different values.

The following experiments compare the accuracy of three approaches. We choose 10 services including services #35-#40 and other 4 services selected randomly from dataset. The comprehensive evaluations of other 4 services should be worse than service #36 that has been identified as the optimal CS among services #35-#40 in the previous experiments. The individual users are still selected from the above 4 ASs. The initial thresholds is set as: P = {0.20, 0.20, 0.20, 0.20}, Q = {0.10, 0.10, 0.10, 0.10}, V = {0.30, 0.30, 0.30, 0.30}. Every experiment is repeated 100 times and check whether three approaches pick out the optimal service #36 correctly. The accuracy is defined as the hit rate of the optimal service #36 in 100 trials. The veto threshold is updated from 0.30 to 0.65 and the above experiment is repeated 8 rounds. The results are shown in Fig. 4.

Fig. 4

Comparative analysis.

From the comparison analysis, it is evident that the proposed approach is more accurate in solving the MCDM-based CS recommendation problems for an SME compared to other two approaches. The major advantages of the proposed approach are as follows: First, it is capable to select the highly trustworthy CS based on the fuzzy data in an uncertain cloud environment by utilizing the trustworthiness INNs to evaluate the performance and potential risks of CSs. Second, it can help those users sensitive to the QoS select valid and satisfactory CSs by identifying the preference, indifference and veto thresholds. Third, by modelling the client contexts as evaluation criteria and exploiting the improved ELECTRE III method, it becomes an effective non-compensatory MCDM approach for SMEs-oriented CS recommendation.

6.4 Execution time analysis based on real dataset

To verify the feasibility of the proposed approach, we analyze its execution time when the more candidate services are selected randomly from the dataset in experiments. The experiments are executed in a notebook with Intel i7-8550U processor and 16G memory. The results are shown in Fig. 5.

Fig. 5

Execution time analysis.

Figure 5 demonstrates that the proposed approach only consumes about 0.5 second when the number of candidate services is up to 40. Thus, the proposed approach explores a complete order of candidate CSs at the low cost of ranking computation in a convenient and practical way.

7 Conclusion

In an uncertain cloud environment, the compensability of evaluation criteria may result in an invalid or unsatisfactory candidate being mistakenly identified as the optimal CS for an SME, which will significantly reduce the accuracy of trustworthy CS recommendation. Aiming at the deficiency of traditional MCDM-based approaches with the compensable decision criteria, this paper proposes a novel client context-aware approach to select trustworthy CSs for SMEs based on non-compensatory MCDM. In this approach, a type of client context is viewed as an independent evaluation criterion, and the INNs are employed to measure the fuzzy trustworthiness of CSs. The outranking relations of INNs are proposed based on theoretical proofs. To solve this problem, an improved ELECTRE III method supporting the trustworthiness INNs is developed by simplifying the process of ranking computation for a total order of all CSs and redefining the indifference, preference and veto thresholds with the relative ratio values. These improvements facilitate increasing the accuracy of SMEs-oriented trustworthy CS recommendation. The case analysis and the experiments based on a real-world dataset demonstrate that the proposed approach can efficiently produce accurate ranking results of candidate CSs, and effectively solve trustworthy CS recommendation problem for an SME.

Footnotes

Appendix

A.1 Original data used in the experiments

The original data used in the experiments is shown in Table A.1.

Performance cloud models and potential risk cloud models

The preprocessed performance cloud models and potential risk cloud models are shown in Tables A.2 and A.3.

Acknowledgment

This work was partially supported by the MOE (Ministry of Education in China) Humanities and Social Science Research Project ("decision-making research on trustworthy cloud service selection for small and medium-sized enterprise users in big data environment", No. 18YJCZH124), Scientific Research Fund of Hunan Provincial Education Department (Outstanding Young Project) (No. 18B037), Natural Science Foundation of Hunan Province, China (No. 2020JJ4440), National Natural Science Foundation of China under Grant (No. 62077014). (Corresponding author: Hongyu Zhang.)

References

Gokalp

M.O.

, Kayabay

, Akyol

M.A.

, Eren

P.E.

and Kocyigit

, Big data for industry 4.0: A conceptual framework, Proceedings of International Conference on Computational Science & Computational Intelligence, IEEE Publisher, (2017), 431–434.

Hochscheid

and Halin

, Generic and SME-specific factors that influence the BIM adoption process: An overview that highlights gaps in the literature, Frontiers of Engineering Management 7 (2020), 119–130.

National Development and Reform Commission of China, Plan on promoting the development of small and medium-sized enterprises (2016–2020), 2021-06-09. url: https://www.ndrc.gov.cn/fggz/fzzlgh/gjjzxgh/201706/t20170620_1196809.html.

Hosseini

, Fallon

, Weerakkody

and Sivarajah

, Cloud computing utilization and mitigation of informational and marketing barriers of the SMEs from the emerging markets, International Journal of Information Management 46 (2019), 54–69.

, Tang

, Zhu

and Zhang

, Resource utilization-aware collaborative optimization of IaaS cloud service composition for data-intensive applications, IEEE Transactions on Systems Man Cybernetics: Systems 51(2) (2021), 1322–1333.

Ibrahim

, Iqbal

M.A.

, Aleem

and Islam

M.A.

, SIM-Cumulus: An academic cloud for the provisioning of network-simulation-as-a-service (NSaaS), IEEE Access 6 (2018), 27313–27323.

Sun

, Dong

, Hussain

F.K.

, Hussain

O.K.

and Chang

, Cloud service selection: State-of-the-art and future research directions, Journal of Network and Computer Applications 45(10) (2014), 134–150.

, Hu

, Li

and Zhu

, Variation-aware cloud service selection via collaborative QoS prediction, IEEE Transactions on Services Computing (2019), DOI:10.1109/TSC.2019.2895784

, Hu

, Li

and Zhang

, Toward trustworthy cloud service selection: A time-aware approach using interval neutrosophic set, Journal of Parallel and Distributed Computing 96 (2016), 75–94.

10.

, Zhu

, Hu

, Li

and Tang

, Time-aware trustworthiness ranking prediction for cloud services using interval neutrosophic set and ELECTRE, Knowledge-Based Systems 138 (2017), 27–45.

11.

Sun

, Ma

, Zhang

, Dong

and Hussain

F.K.

, Cloud-FuSeR: Fuzzy ontology and MCDM based cloud service selection, Future Generation Computer Systems 57 (2016), 42–55.

12.

Ding

, Liu

, Sun

, Jiang

and Zhou

, A transaction and QoS-aware service selection approach based on genetic algorithm, IEEE Transactions on Systems Man & Cybernetics: Systems 45(7) (2017), 1035–1046.

13.

Strumberger

, Bacanin

, Tuba

and Tuba

, Resource scheduling in cloud computing based on a hybridized whale optimization algorithm, Applied Sciences 9(22) (2019), 4893–4893.

14.

, Zhu

, Hu

, Tang

and Dong

, Multi-valuedcollaborative QoS prediction for cloud service via time seriesanalysis, Future Generation Computer Systems 68(3) (2017), 275–288.

15.

Rosaci

and Sarné

G.M.

, Recommending multimedia web servicesin a multi-device environment, Information Systems 38(2) (2013), 198–212.

16.

Wei

, Yin

, Li

and Wu

, Efficient web service QoS prediction using local neighborhood matrix factorization, Engineering Applications of Artificial Intelligence 38 (2015), 14–23.

17.

, Hu

, Yang

and Song

, User feature-aware trustworthiness measurement of cloud services via evidence synthesis for potential users, Journal of Visual Languages and Computing 25(6) (2014), 791–799.

18.

Choi

C.R.

and Jeong

H.Y.

, A broker-based quality evaluation system for service selection according to the QoS preferences of users, Information Sciences 277 (2014), 553–566.

19.

Mehdi

, Bouguila

and Bentahar

, Probabilistic approach for QoS-aware recommender system for trustworthy web service selection, Applied Intelligence 41(2) (2014), 503–524.

20.

and Hu

, User preferences-aware recommendation for trustworthy cloud services based on fuzzy clustering, Journal of Central South University 22(9) (2015), 3495–3505.

21.

, Zhu

and Wang

, Cloud service selection model based on QoS-aware in cloud manufacturing environment, Computer Integrated Manufacturing Systems 20(5) (2014), 1246–1254.

22.

and Hu

, Recommend trustworthy services using interval numbers of four parameters via cloud model for potential users, Frontiers of Computer Science 9(6) (2015), 887–903.

23.

Wang

, Smarandache

, Zhang

and Sunderraman

, Interval Neutrosophic sets and logic: Theory and applications in computing: theory and applications in computing, Hexis Phoenix Az (2005).

24.

Govindan

and Jepsen

M.B.

, ELECTRE: A comprehensive literature review on methodologies and applications, European Journal of Operational Research 250(1) (2016), 1–29.

25.

Shafia

M.A.

, Rahimi Moghaddam

and Teimoury

, Ranking fuzzy cognitive map based scenarios using ELECTRE III: Applied on housing market, Expert Systems 33(5) (2016), 417–431.

26.

Bottero

, Ferretti

, Figueira

J.R.

, Greco

and Roy

, Dealing with a multiple criteria environmental problem with interaction effects between criteria through an extension of the Electre III method, European Journal of Operational Research 245(3) (2015), 837–850.

27.

Roy

, The outranking approach and the foundations of ELECTRE methods, Theory and Decision 31 (1991), 49–73.

28.

Roy

, Classement et choix en présence de points de vuemultiples, Revue française d’informatique et de rechercheopérationnelle 2(8) (1968), 57–75.

29.

Garg

S.K.

, Versteeg

and Buyya

, A framework for ranking of cloud computing services, Future Generation Computer Systems 29(4) (2013), 1012–1023.

30.

Liu

, Chan

F.T.S.

and Ran

, Decision making for the selection of cloud vendor: An improved approach under group decision-making with integrated weights and objective/subjective attributes, Expert Systems with Applications 55 (2016), 37–47.

31.

Wang

and Wang

, The probability theories for IVFSs and IVIFSs,(12), Computers and Mathematics with Applications 62 (2011), 4535–4538.

32.

, Multicriteria decision-making method using the correlation coefficient under single-valued neutrosophic environment, International Journal of General Systems 42(4) (2013), 386–394.

33.

Smarandache

, Neutrosophy / Neutrosophic probability, set, and logic, ProQuest Info & Learning, Ann Arbor, MI, USA, (1998).

34.

Smarandache

, A unifying field in logics: Neutrosophic logic, American Research Press, Rehoboth, NM, (1999).

35.

Smarandache

, A unifying field in logics: Neutrosophic logic. Neutrosophy, Neutrosophic set, Neutrosophic probability and statistics, ProQuest Info & Learning, Ann Arbor, MI, USA, (2007).

36.

, A multicriteria decision-making method using aggregation operators for simplified neutrosophic sets, Journal of Intelligent & Fuzzy Systems 26(5) (2014), 2459–2466.

37.

Abdel-Basset

, Nabeeh

N.A.

, El-Ghareeb

H.A.

and Aboelfetouh

, Utilising neutrosophic theory to solve transition difficulties of IoT-based enterprises, Enterprise Information Systems 14(9–10) (2020), 1304–1324.

38.

Zavadskas

E.K.

, Bausys

, Kaklauskas

, Ubarte

, Kuzminske

and Gudiene

, Sustainable market valuation of buildings by the single-valued neutrosophic MAMVA method, Applied Soft Computing 57 (2017), 74–87.

39.

Thanh

N.D.

and Ali

, A novel clustering algorithm in a Neutrosophic recommender system for medical diagnosis, Cognitive Computation 9(4) (2017), 526–544.

40.

, Zhang

and Wang

, A fuzzy decision support model with sentiment analysis for items comparison in e-commerce: The case study of PConline.com, IEEE Transactions on Systems Man and Cybernetics: Systems 49(10) (2019), 1993–2004.

41.

Zhang

, Ji

, Wang

and Chen

, An improved weighted correlation coefficient based on integrated weight for interval neutrosophic sets and its application in multi-criteria decision-making problems, International Journal of Computational Intelligence Systems 8(6) (2015), 1027–1043.

42.

Liu

, Chu

, Li

and Chen

, Some generalized neutrosophic number Hamacher aggregation operators and their application to group decision making, International Journal of Fuzzy Systems 16(2) (2014), 242–255.

43.

Karaaslan

, Multicriteria decision-making method based on similarity measures under single-valued neutrosophic refined and interval neutrosophic refined environments, International Journal of Intelligent Systems 33(5) (2018), 928–952.

44.

Peng

, Wang

and Yang

, A multi-valued neutrosophic qualitative flexible approach based on likelihood for multi-criteria decision-making problems, International Journal of Systems Science 48(2) (2017), 425–435.

45.

and Zhang

, An overview on the applications of the hesitant fuzzy sets in group decision-making: Theory, support and methods, Frontiers of Engineering Management 6(2) (2019), 163–182.

46.

Wang

, Xu

and Li

, Generic normal cloud model, Information Sciences 280 (2014), 1–15.

47.

Zhao

, Dong

, Chen

, Jiang

and Dong

, The combined cloud model for edge detection, Multimedia Tools and Applications 76(13) (2017), 15007–15026.

48.

Peng

, Zhang

and Wang

, Cloud decision support model for selecting hotels on TripAdvisor.com with probabilistic linguistic information, International Journal of Hospitality Management 68 (2018), 124–138.

49.

Chinese University of Hong Kong, WS-DREAM, 2021-1-19. url: https://wsdream.github.io/.

50.

Tang

, Dai

, Liu

and Chen

, Towards a trust evaluation middleware for cloud service selection, Future Generation Computer Systems 74 (2017), 302–312.