Applying interrater reliability measure for user credibility assessment in reputation-oriented service discovery

Abstract

Service Oriented Architecture is built on services in which web service discovery is one of the most widely explored domains. Service reputation plays a vital role in the discovery process while selecting the optimal service from a large pool of functionally equivalent services. Reputation mechanism attempts to forecast the future performance of a service based on its past behaviors generally obtained in the form of feedback ratings submitted by users of the service. These feedback ratings about a service may vary from user to user. Variations in feedback ratings could be because of different users’ different subjective judgments and/or dishonest users’ purposely submitted unfair ratings. This paper proposes a service reputation measurement approach counting such diversified act of rating. To realize this goal, an efficient user credibility assessment methodology has been devised by employing the measure of interrater reliability. Experiments are performed to validate the feasibility and effectiveness of the proposed reputation measurement approach. The experimental results depict that the proposed approach can fairly assess service reputations in the presence of various kinds of raters in the system.

Keywords

Service reputation user credibility subjective feedback rating malicious feedback rating interrater reliability Cohen’s weighted Kappa Spearman’s rank-order correlation coefficient

1. Introduction

In Service Oriented Architecture (SOA), with the invent of Web services, the Web has become a platform where applications, built based on services, can be automatically invoked by other Web clients [10]. W3C defines web service as: “A Web service is a software system designed to support interoperable machine-to-machine interaction over a network. It has an interface described in a machine-processable format (specifically WSDL). Other systems interact with the Web service in a manner prescribed by its description using SOAP messages, typically conveyed using HTTP with an XML serialization in conjunction with other Web-related standards” [2]. Web Services architecture authorizes interactions among three entities: service provider, service consumer, and service registry [8].

Due to the popularity of SOA, services over the Web are growing rapidly. This has resulted in a number of services offering similar functionality [27]. Therefore, selecting the optimal service from a large pool of functionally equivalent services has become a challenging task. Since the providers of the services may not perform according to what have been advertised by them [11], care must be taken while selecting a suitable service that satisfies the consumer’s need. A service selection mechanism must not select any service that harms a consumer’s interest. The selection mechanism must explore ‘how well a service will do’ along with ‘what a service can do’. To achieve this, service reputation assessment plays a vital role in discovery process.

A reputation mechanism attempts to forecast the future performance of a service based on its past behaviors [21]. Service reputations speak about the degree of competency of services in the competitive service market. Almost all reputation oriented service discovery mechanisms utilize the feedback ratings left by the service consumers after availing services to evaluate the service reputations. The feedback rating generally portrays the level of satisfaction of a consumer’s experience with an invoked service. In ideal situations, the feedback ratings of different consumers about a service must comply with each other. However, in real-world environment, the feedback ratings submitted for a particular service may vary from consumer to consumer due to several reasons. Two such reasons can be (i) different consumers’ different subjective judgments, and (ii) dishonest consumers’ purposely submitted unfair ratings. It is a fact that different consumers of different mindset exist in online open environment and these consumers may not always carry out the act of rating in an identical fashion. They may exercise their subjective preferences while reporting their feedback ratings about the perceived service quality. On the other hand, the existence of dishonest consumers in such open environment cannot be overlooked. These consumers submit varying ratings that are not in accordance with the deserving ratings of a service. They do it purposely in order to upgrade or downgrade the overall reputation of the service in the business market [1,5]. Thus, counting the diversified act of rating, there is a strong need of devising an efficient methodology at this point to counter such situations and to support the genuine services to maintain high reputation scores in this competitive business market.

Previous approaches [12,13,25] in service reputation measurement dealt only with the presence of dishonest consumers while devising a methodology for user credibility assessment. In contrast, the proposed reputation measurement approach takes into account the presence of both dishonest consumers and consumers with subjective preferences in the system. Wherein, an efficient user credibility assessment methodology has been devised by employing the measure of interrater reliability. Interrater reliability is the measure of the degree of similarity between the reported feedback ratings of consumers (raters) of a service. Application of interrater reliability measure in user credibility assessment ensures the appropriate reflection of credibility of each consumer in reporting feedback ratings at par other consumers of the system. And then, the user credibility aids in evaluating the reputations of the services. This summarizes the contribution of this paper.

The remaining sections of this paper are organized as follows: Section 2 reviews the related works. Section 3 presents the proposed reputation measurement approach. In Section 4, the experimental results based on simulated scenarios are discussed. Section 5 concludes the paper.

2. Related work

In this section, prominent service reputation measurement approaches existing in the literature are discussed.

Malik and Bouguettaya [10,12] emphasized on shielding of reputation based service discovery system from unfair or malicious ratings. They proposed a methodology for evaluating the credibility of consumers/raters thereby assessing the reputation of providers by assigning more weight to honest raters’ and relatively less weight to dishonest raters’ ratings. Their approach is based primarily on the concept of majority rating scheme in which a particular consumer’s credibility is increased or decreased depending upon the degree of correspondence of the feedback rating provided by the consumer with the majority opinion. They formulated the overall credibility of a consumer as a function of three parameters: the historical behavior of the consumer, the agreement/disagreement of the rating provided by the consumer with the majority rating, the equivalence of the rating provided by the consumer with the last evaluated reputation value of the provider. In ‘RATEWeb’ [12], the reputation assessment framework also caters additional metrics for reputation evaluation viz personalized preferences, temporal sensitivity, and first-hand interaction knowledge.

Hien Trang Nguyen et al. [13] proposed a trust and reputation system for web service selection based on Bayesian network model. The overall reputation of a service is derived based on the weighted integration of three different sources of trust: direct experience opinion of the requester (direct trust), recommendation from other consumers (recommendation trust, subjective view) and QoS monitoring information (conformance trust, objective view). The credibility of a consumer as a rater, in providing recommendations, is calculated as the weighted integration of two factors: usefulness of the rater’s feedbacks (global measure) and similarity of ratings and QoS preferences between the rater and requester (personalized measure).

Yan Wu et al. [25] proposed a two-phase approach to assess the service reputations. To capture the variations in service reputations over time due to change in service quality, a dynamic weight formula (DWF) is proposed in the first phase (when the ratings are believed to be fair) to reflect the current nature of service behavior. To deal with false variations in service reputations because of the effect of unfair ratings submitted by dishonest consumers, the authors in the second phase suggested the use of olfactory response formula (ORF) to suppress the negative impacts of malicious ratings.

In [28], the reputation of a service is assessed as the weighted integration of direct experience based reputation and recommendation based reputation. The recommendation based reputation about a service for a consumer is both similarity (between requester and recommender) and trust (on recommender from requester) weighted average of recommended reputations of the service provided by other consumers.

In [23], the service reputation measurement approach consists of two phases: (i) malicious feedback rating detection to identify unfair ratings purposely submitted by dishonest users by employing cumulative sum (CUSUM) control chart, and (ii) feedback rating adjustment by computing feedback similarity employing Pearson Correlation Coefficient (PCC) to deal with different preferences the users exhibit while reporting their feedbacks.

Talal H. Noor et al. [14] designed CloudArmor (CLOud consUmers creDibility Assessment & tRust manageMent of clOud seRvices) which is a framework to support reputation-based trust management of services in cloud environments. It includes (i) a Zero-Knowledge Credibility Proof Protocol (ZKC2P) that enables the Trust Management Service (TMS) to prove the credibility of consumer feedbacks and preserves consumers’ privacy, (ii) feedback credibility assessment model to shield the services from malicious consumers and to evaluate trustworthiness of the services, (iii) an availability model to ensure the availability of decentralized implementation of TMS. The trust result (reputation) of a service is evaluated based on its all attained trust feedbacks weighted by their corresponding credibility scores, the total number of trust feedbacks and the change rate of trust results of the service during a time period.

[29] proposed a domain-partition based trust model to minimize trust storage and computing overhead and maximize the ability to detect malicious nodes and their unfaithful trust evaluations in unreliable cloud environments. Domain and cross-domain sliding-windows are designed to store the most recent trust values. Then, an algorithm is developed to compute domain and cross-domain trust values for nodes, and a filter procedure is adopted to remove malicious trust evaluations and malicious nodes from a domain.

Miao Wang et al. [22] presented a High-reliability Multi-faceted Reputation (HMRep) evaluation methodology for online services. In the first phase, they addressed and estimated the missing feedback ratings of various services based on rating behavior of the user and quality of the service. To efficiently assess the service reputation, their model finds and then eliminates malicious and irresponsible consumers from the system in the second step. And, then the service reputation is assessed based on its received feedback ratings and their corresponding consumers’ credibility. In order to properly reflect the change in service quality, their model also makes use of historical information of the services during service reputation calculation.

Okba Tibermacine et al. [18] proposed a three-phase process for evaluating web service reputation. The process applies at two rounds an adapted HITS (Hypertext Induced Topic Search) algorithm to evaluate the user credibility values and service reputation values. In first phase, the credibility values of the users are evaluated based on the majority voting model. In second phase, an analysis of the dispersion of user credibility values is done to identify a threshold that separates honest and malicious users. Then, a refined model excluding malicious users and their feedback ratings is constructed. In the last phase, a second round of HITS algorithm is applied for the final evaluation of service reputation.

The following two observations are drawn after analyzing the literature:

Many of the reputation measurement approaches have adopted the majority rating scheme while devising a methodology for user credibility assessment. In such scheme, the user credibility score increases if the consumer’s reported feedback rating matches with the majority opinion and decreases otherwise. Consumers of the majority group attain high user credibility scores; while other consumers attain relatively low user credibility scores based on the amount of deviations of the feedback ratings from the majority opinion. Systems, which rely on majority rating scheme, believe that majority of the consumers are honest and report trustworthy feedback ratings. However, the majority rating scheme may sometimes fail to accurately measure the service reputations if malicious consumers form the majority group or the majority raters collude. Under such circumstances, the opinion of the majority group formed by malicious consumers will adversely influence the user credibility scores of different consumers which, in turn, will affect the fair feedback ratings of honest consumers.

The reputation measurement approaches existing in the literature consider two types of consumers/raters: honest and malicious. Honest consumers report feedback ratings at par the perceived service quality. On the other hand, malicious consumers purposely submit distorted feedback ratings about the perceived quality. However, another type of consumers also exists in the system who provide subjective judgments [19], but are not explored by the research community. This type of raters is also honest in nature, but exercise their subjective preferences while reporting feedback ratings about the perceived service quality. Therefore, if not treated separately, they will also be considered as malicious consumers as their fair feedback ratings can vary depending on their subjective preferences. Thus, these consumers will be assigned unjustifiable user credibility scores as compared to the honest consumers.

Owing to the above facts, a user credibility assessment methodology has been proposed in this paper which

avoids dependency on majority opinion during the assessment of user credibility scores.

considers the existence of consumers with subjective judgments along with honest and malicious consumers.

3. Reputation measurement approach

3.1. Preliminaries

3.1.1. User credibility assessment

One of the most important factors that must be addressed for the success of reputation-oriented service discovery in feedback based systems is the efficient assessment of user credibility. In online open environment, where user activities are mostly vague, user credibility plays a vital role in knowing service reputation better. User credibility represents the quality of being trustworthy in providing feedback rating [11]. It is a good practice to give more weightage to the feedback ratings of high credible users and relatively less weightage to the feedback ratings of low credible users [7,26].

Reputation-oriented service discovery relies on feedback based models and the rating scale is considered to be the most prominently used measuring instrument utilized by the consumer to express its level of satisfaction about the perceived quality of a service by selecting some point on the scale. This aids in obtaining sets of feedback ratings for different services, which have been actively availed by the consumers of the system. These feedback ratings about a service may vary from consumer to consumer due to variety of reasons. Two such reasons, on which the proposed model intends to concentrate simultaneously in this paper, are: (i) different consumers’ different subjective judgments about the same service, (ii) dishonest consumers’ malicious ratings.

Though feedback ratings are considered to be the rich sources of data to express satisfaction levels about perceived service quality, they might be subject to various sources of bias and error. As there is a lack of well-guided systematic process for consumers of the system to follow while reporting their feedback ratings, it is obvious that these consumers may not carry out the act of rating in an identical fashion. In online open environment, different consumers of different mindset, residing in distributed locations, can arbitrarily join the system. Therefore, it is not an easy job to bring them all in a common platform to train them on how to use the rating scale and how to perform the act of feedback rating for different qualities of services they might experience while availing the services in real time.

A consumer may appraise the performance of a service to be satisfactory. Another consumer may report the same performance of the same service as moderate. While, some other consumer may find it to be unsatisfactory. Though the consumers have exercised their feedback ratings honestly, the disparities in their judgments about the same service may have occurred because of the consumers’ varying satisfaction levels which resulted in the selection of different points on the rating scale. It is worth noting that subjective judgments may differ from consumer to consumer and contain high degree of personal preferences. So, the system must accept that consumers of such open environment are heterogeneous in nature and they might exercise their subjective preferences while reporting their feedback ratings about the perceived service quality.

A consumer might not give much effort in choosing the appropriate point on the rating scale in accordance with the perceived service quality. The consumer might arbitrarily select any point on the scale only for the sake of performing the act of feedback rating. Some of the probable reasons could be that the consumer is unaware of the use of the rating scale, or unable to differentiate between the relative points on the rating scale, or does not actually know which point on the rating scale is to be chosen for the perceived service quality. In this way, some consumers introduce some errors in the feedback rating data [15].

In addition, the existence of dishonest consumers in such open environment cannot be overlooked. These consumers submit varying ratings that are not in accordance with the deserving ratings of a service. They do it purposely in order to upgrade or downgrade the overall reputation of the service in the business market [1,5]. Several other unknown, irrelevant factors also influence the judgment process that can distort the validity of feedback ratings leading towards erroneous evaluation of service reputation.

Counting all these facts under a single umbrella and supposing that for each service, every datum recorded on the rating scale is the result of either subjective judgment of the consumer or honest rating or malicious rating or rating error, the generalization of each service’s set of feedback ratings is important to precisely evaluate the service reparation. To realize this goal, the measure of interrater reliability or interrater agreement is employed to devise a methodology for user credibility assessment.

3.1.2. Interrater reliability

Interrater reliability or agreement is the measure of the degree of similarity between the reported feedback ratings of consumers (raters) of a service. It defines how much consensus or consistency is there in the feedback ratings reported by various consumers as raters. In the proposed model, the interrater consensus and consistency estimates of interrater reliability are used to deal with the assessment of user credibility of individual consumers/users. Application of these two interrater reliability measures in user credibility assessment ensures the appropriate reflection of credibility of each consumer in reporting feedback ratings. The interrater consensus estimate represents the extent to which different consumers tend to report exactly the same feedback ratings about the rated service. On the other hand, interrater consistency estimate represents the degree to which the feedback ratings of different consumers are proportional when expressed as deviations from their means [19]. In practice, the interrater consistency estimate represents the extent to which different consumers tend to report the same relative ordering or ranking of the rated service. In a nutshell, interrater consensus estimate refers to the absolute correspondence of feedback ratings, while interrater consistency estimate refers to the relative correspondence of feedback ratings. It is to be noted that low interrater consensus and high interrater consistency (or vice versa) may exist simultaneously. For instance, one consumer’s reported feedback ratings about services may be consistently one or two scale points higher than another consumer’s reported feedback ratings towards the same services. In this example, a high estimate of interrater consistency is observed since the relative ordering of the services for both the consumers are much same, though, exact agreement between these consumers has never been achieved which resulted in low interrater consensus.

3.2. Proposed framework

3.2.1. Model setup

A service provider can offer any number of services; but for simplicity, this model assumes that a service provider offers only one service. The terms service provider and service are used interchangeably throughout this paper. Similarly, the terms service consumer, user and rater are also used interchangeably in this paper. The model assumes that m number of consumers and n number of services reside in the system. The set of consumers is denoted by $C = {c_{1}, c_{2}, \dots, c_{m}}$ and the set of services is denoted by $S = {s_{1}, s_{2}, \dots, s_{n}}$ . A consumer, after availing a service, reports its level of satisfaction about the perceived quality of the service in the form of feedback rating. Let $r_{i, j}$ denotes such feedback rating from consumer $c_{i}$ to service $s_{j}$ .

3.2.2. Computing consensus estimate

To compute consensus estimate of interrater reliability, the weighted Kappa coefficient of Cohen [4] has been adopted. In estimating the degree of consensus between two consumers/raters, Cohen’s Kappa (κ) [3] is a measure of “the proportion of agreement after chance agreement is removed from consideration” and is mathematically calculated as given in equation (1). $\begin{matrix} (1) & κ = \frac{P_{o} - P_{c}}{1 - P_{c}} \end{matrix}$ where, $P_{o}$ is the observed proportion of agreement, and $P_{c}$ is the proportion of agreement expected by chance.

κ treats all kinds of disagreements (either by 1 scale point or by more) between any two raters equally, i.e. as total disagreement. But, situations may arise when certain disagreements between two raters are to be penalized more (or less) than others. Thus, to cater for partial agreements, Cohen developed weighted Kappa ( $κ_{w}$ ) [4], to measure the proportion of weighted agreement corrected for chance. Cohen’s weighted Kappa ( $κ_{w}$ ) permits the assignment of a weight of 1 to all agreements and varying weights ([0,1[) to various levels of disagreements. Both linear and quadratic weighting schemes are available; the quadratic weighting scheme has been used in the proposed model for the purpose of weighing different levels of disagreements between two raters.

Since, all consumers may not avail all services of the system, complete feedback ratings of the services may not always be available for any two consumers. K. L. Gwet [6] recommends a general formulation of Cohen’s weighted Kappa ( $κ_{w}$ ), considering the fact of missing ratings, which is equally adopted in the proposed model in order to compute the consensus estimate between any two consumers/raters, say $c_{a}$ and $c_{b}$ . Let the consumers report their feedback ratings for services on a q-point $(1, \dots, f, \dots, q)$ rating scale as defined by the system. Following parameter values are to be known in calculating $κ_{w}$ . $\begin{matrix} (2) & P_{f g}^{'} = \frac{P_{f g}}{(1 - P_{a \oplus b})} \end{matrix}$ where, $P_{a ⊕ b}$ is the relative number of services rated by a single consumer (either consumer $c_{a}$ or consumer $c_{b}$ , but not both), and $P_{f g}$ is the relative number of services classified into categories f and g by consumers $c_{a}$ and $c_{b}$ respectively. $\begin{matrix} (3) & P_{f}^{'} = \frac{P_{f}}{(1 - P_{\sim a})} \end{matrix}$ where, $P_{f}$ is the relative number of services classified into category f by consumer $c_{a}$ , and $P_{\sim a}$ is the relative number of services not rated by consumer $c_{a}$ . $\begin{matrix} (4) & P_{g}^{'} = \frac{P_{g}}{(1 - P_{\sim b})} \end{matrix}$ where, $P_{g}$ is the relative number of services classified into category g by consumer $c_{b}$ , and $P_{\sim b}$ is the relative number of services not rated by consumer $c_{b}$ . $\begin{matrix} (5) & w_{f g} = \{\begin{matrix} 1 - \frac{{(f - g)}^{2}}{{(q - 1)}^{2}}, & if f \neq g \\ 1, & if f = g \end{matrix} \end{matrix}$ where, $w_{f g}$ denotes the weight to be associated with any two categories f and g, and is derived employing quadratic weighting scheme.

Thus, the weighted Kappa coefficient of Cohen ( $k_{w}$ ) is given by equation (6). $\begin{matrix} (6) & κ_{w} = \frac{P_{o} - P_{c}}{1 - P_{c}} \end{matrix}$ where, $\begin{matrix} (7) & P_{o} = \sum_{f, g}^{q} w_{f g} * P_{f g}^{'} \end{matrix}$ and, $\begin{matrix} (8) & P_{c} = \sum_{f, g}^{q} w_{f g} * P_{f}^{'} * P_{g}^{'} \end{matrix}$

A value for $κ_{w}$ lies in the range $[- 1, 1]$ ; where, $κ_{w} = 1$ specifies perfect agreement; $κ_{w} = 0$ specifies that the observed agreement is equal to the expected chance agreement; and, $κ_{w} < 0$ specifies that observed agreement is less than expected chance agreement [20]. The widely accepted interpretations of various $κ_{w}$ values as suggested by Landis and Koch [9] are shown in Table 1.

Table 1
Interpretations of various $κ_{w}$ values as suggested by Landis and Koch

Value of $κ_{w}$ Strength of Agreement

<0.00 Poor

0.00–0.20 Slight

0.21–0.40 Fair

0.41–0.60 Moderate

0.61–0.80 Substantial

0.81–1.00 Almost Perfect

Value of $κ_{w}$	Strength of Agreement
<0.00	Poor
0.00–0.20	Slight
0.21–0.40	Fair
0.41–0.60	Moderate
0.61–0.80	Substantial
0.81–1.00	Almost Perfect

The consensus estimate between two consumers/raters, $c_{a}$ and $c_{b}$ , computed using Cohen’s weighted Kappa ( $κ_{w}$ ), is denoted by $c s (c_{a}, c_{b})$ in the proposed model. The model adopts Landis and Koch’s interpretation of $κ_{w}$ values [9], and make all $c s (c_{a}, c_{b})$ ( $a, b = 1$ to m, and $a \neq b$ ) equal to 0.00 if they are found to be negative, so as to reduce their effect in the computation of consumer/user credibility, i.e. $c s (c_{a}, c_{b}) = 0.00$ , if $c s (c_{a}, c_{b}) < 0.00$ .

3.2.3. Computing consistency estimate

To compute consistency estimate of interrater reliability, the Spearman’s rank-order correlation coefficient (denoted by ρ (rho) or $r_{s}$ ) [16] has been adopted, which is a nonparametric measure of correlation between the rankings of two variables. In this model, the Spearman’s rank-order correlation coefficient has been used to measure the degree of correspondence between two raters in ranking the services reflected from their reported feedback ratings. The most obvious advantage of using Spearman’s rho is that it does not impose any restriction on the distribution of data under investigation.

Let $c y (c_{a}, c_{b})$ denotes the consistency estimate between two consumers/raters, $c_{a}$ and $c_{b}$ , which is computed using Spearman’s rank-order correlation coefficient as given in equation (9). Here, ${CRS}_{a b} \subseteq S$ denotes the set of commonly rated services by both consumers/raters $c_{a}$ and $c_{b}$ , and $x ⩽ n$ be the size of ${CRS}_{a b}$ . $\begin{matrix} (9) & c y (c_{a}, c_{b}) = 1 - \frac{6 \sum_{s_{j} \in {CRS}_{a b}} d_{j}^{2}}{x (x^{2} - 1)} \end{matrix}$ where, $d_{j} = r_{a, j} - r_{b, j}$ denotes the difference between the ranks of service $s_{j} \in {CRS}_{a b}$ as reported by the two consumers/raters $c_{a}$ and $c_{b}$ .

Equation (9) is useful in situations when ranks of services as observed by any of the consumers are all different. However in reality, situations of tied ranks may occur when more than one service are assigned the same rank by a consumer. In such situations, services having tied ranks are assigned a common rank which is the average of the ranks of positions they occupy in the ordered service set of an individual consumer. Thus, taking tied ranks into account, the following version of Spearman’s rank-order correlation coefficient, given by equation (10), has been used in the proposed model to compute the interrater consistency estimate, $c y (c_{a}, c_{b})$ . $\begin{matrix} (10) & c y (c_{a}, c_{b}) = 1 - \frac{6 {\sum_{s_{j} \in {CRS}_{a b}} d_{j}^{2} + \frac{(T_{a} + T_{b})}{12}}}{x (x^{2} - 1)} \end{matrix}$ where, $\begin{matrix} (11) & T_{a} = \sum_{i = 1}^{g} t_{i} (t_{i}^{2} - 1) \end{matrix}$ and, $\begin{matrix} (12) & T_{b} = \sum_{i = 1}^{g} t_{i} (t_{i}^{2} - 1) \end{matrix}$ where, g is the number of tied groups of services for that consumer and $t_{i}$ is the number of services with tied rank in a tie group. If there are no services with tied rank for a consumer, then $T = 0$ .

A value for $r_{s}$ also lies in the range $[- 1, 1]$ ; where, 1 indicates a perfect association, 0 indicates no association, and −1 indicates a perfect negative association between the consumers/raters. The closer $r_{s}$ is to 0, the weaker the association between the consumers/raters. The proposed model focuses on positive associations rather than negative associations. In this model, the negative association is assumed to carry the similar notion to that of zero association. Therefore, similar to interrater consensus estimate, the model makes all $c y (c_{a}, c_{b})$ ( $a, b = 1$ to m, and $a \neq b$ ) equal to 0.00 if they are found to be negative, i.e. $c y (c_{a}, c_{b}) = 0.00$ , if $c y (c_{a}, c_{b}) < 0.00$ .

3.2.4. Computing consensus and consistency estimates of single rater vs rest of raters

Until now, the model has computed the consensus estimate, $c s (c_{a}, c_{b})$ , and consistency estimate, $c y (c_{a}, c_{b})$ , of every pair of raters, $c_{a}$ and $c_{b}$ ( $a, b = 1$ to m, and $a \neq b$ ), in the system. The next task is to compute the consensus and consistency estimates of single rater vs rest of raters, denoted by $c s_{sr_ror} (c_{a}, C_{rest})$ and $c y_{sr_ror} (c_{a}, C_{rest})$ respectively, where $C_{rest} = C - c_{a}$ .

The consensus estimate of single rater vs rest of raters, $c s_{sr_ror} (c_{a}, C_{rest})$ , is computed by averaging the pairwise consensus estimates of the rater $c_{a}$ with every other rater $c_{b} \in C_{rest}$ and is given by equation (13). $\begin{matrix} (13) & c s_{sr_ror} (c_{a}, C_{rest}) = \frac{\sum_{c_{b} \in C_{rest}} c s (c_{a}, c_{b})}{m - 1} \end{matrix}$ where, m is the total number of consumers/raters in the system.

Similarly, the consistency estimate of single rater vs rest of raters, $c y_{sr_ror} (c_{a}, C_{rest})$ , is computed by averaging the pairwise consistency estimates of the rater $c_{a}$ with every other rater $c_{b} \in C_{rest}$ and is given by equation (14). $\begin{matrix} (14) & c y_{sr_ror} (c_{a}, C_{rest}) = \frac{\sum_{c_{b} \in C_{rest}} c y (c_{a}, c_{b})}{m - 1} \end{matrix}$

A value for $c s_{sr_ror} (c_{a}, C_{rest})$ or $c y_{sr_ror} (c_{a}, C_{rest})$ lies in the range $[0, 1]$ ; where, higher and lower values indicate strong and weak consensus or consistency estimate of single rater vs rest of raters, respectively.

3.2.5. Computing user credibility

Both $c s_{sr_ror} (c_{a}, C_{rest})$ and $c y_{sr_ror} (c_{a}, C_{rest})$ of each user/consumer, $c_{a}$ ( $a = 1$ to m), will contribute every information in assessing its user credibility, ${UC}_{a}$ . The user credibility, ${UC}_{a}$ , of a user, $c_{a}$ , is computed as the weighted summation of its consensus estimate of single rater vs rest of raters and consistency estimate of single rater vs rest of raters. That is, $\begin{matrix} (15) & {UC}_{a} = c s_{w} * c s_{sr_ror} (c_{a}, C_{rest}) + c y_{w} * c y_{sr_ror} (c_{a}, C_{rest}) \end{matrix}$ where, $c s_{w}$ and $c y_{w}$ denote the weighting factors for consensus and consistency estimates of single rater vs rest of raters, respectively. $c s_{w}$ , $c y_{w} = [0, 1]$ and $c s_{w} + c y_{w} = 1$ . The choice of values for $c s_{w}$ and $c y_{w}$ is left at the discretion of the consumer who wishes to avail a service. However, if the requesting consumer blurs the distinction between these two weighting factors, $c s_{w}$ and $c y_{w}$ , the system will take up similar values for both of them. Equation (15) also ensures the user credibility, ${UC}_{a}$ ( $a = 1$ to m), to range in the scale $[0, 1]$ , where 1 indicates the most trustworthy user and 0 indicates a complete dishonest user.

3.3. The service reputation calculation

Having calculated the user credibility, ${UC}_{a}$ , of each user, $c_{a}$ ( $a = 1$ to m), the system is ready to evaluate the reputation of each service, $s_{j}$ ( $j = 1$ to n), denoted as REPU( $s_{j}$ ). It is calculated using equation (16). $\begin{matrix} (16) & REPU (s_{j}) = \frac{\sum_{c_{a} \in {UA}_{j}} r_{a, j} * {UC}_{a}}{\sum_{c_{a} \in {UA}_{j}} {UC}_{a}} \end{matrix}$ where, ${UA}_{j}$ denotes the set of users who have availed the service $s_{j}$ . Equation (16) can be described as: the reputation of service $s_{j}$ , REPU( $s_{j}$ ), is the user credibility, ${UC}_{a}$ ( $c_{a} \in {UA}_{j}$ ), weighted average of its all attained feedback ratings, $r_{a, j}$ .

4. Experiments and results

Experiments are performed to validate the feasibility and effectiveness of the proposed reputation measurement approach and the results, thus obtained, are reported in this section.

Due to the current limited availability of feedback rating data in service-oriented environment, many existing service reputation measurement approaches [12,17,18,23,25] dealt with simulation data for the purpose of performance evaluation of their models. Therefore, to evaluate the performance of the reputation measurement approach proposed in this paper, the experiments are performed on a simulated environment in which various kinds of consumers interact with the services to generate various kinds of feedback ratings.

4.1. Experimental setup

The simulated environment consists of 100 services and 200 consumers. Services with different performance levels ( $PerfLev \in [1, 10]$ ) exist in the system. Each performance level reflects the actual measure of quality delivered by a service on a scale of 10. Though in reality, the actual performances of the services can never be accessed; but for experimental purpose, they need to be captured. The service pool is divided into ten groups of ten members each. Services in each group share the same performance level and each group corresponds to a distinct performance level. However, the actual performance levels of the services are not known to the proposed reputation evaluator. The reputation evaluator estimates the reputations of the services (EstRepu) based on the feedback ratings submitted by different consumers. The Root Mean Square Errors (RMSEs) between the actual performance levels and the estimated reputations of the services are reported to validate the proposed reputation measurement approach. $\begin{matrix} (17) & RMSE = \sqrt{\frac{\sum_{j = 1}^{n} {(PerfLev (s_{j}) - EstRepu (s_{j}))}^{2}}{n}} \end{matrix}$ where, $n = 100$ denotes the size of the service pool.

Consumers/raters also report their feedback ratings about services on a 10-point integer rating scale. Three types of raters exist in the simulated environment: RaterType#1, RaterType#2, and RaterType#3.

RaterType#1: Raters of this type are honest in nature. They always submit the feedback ratings at par the perceived service quality. That is, their submitted feedback ratings always conform to the actual performance levels, PerfLev, of the services.

RaterType#2: Raters of this type are also honest in nature, but always exercise their subjective preferences while reporting their feedback ratings about the perceived service quality. The feedback ratings of this type of raters are simulated with a deviation of $\pm [1, 2]$ from PerfLev. That is, they always generate random feedback ratings in the range $[max (1, PerfLev - 2), min (PerfLev + 2, 10)]$ excluding PerfLev.

RaterType#3: Raters of this type are dishonest or malicious in nature. They always purposely submit distorted feedback ratings about the perceived service quality. This type of raters is simulated to generate random feedback ratings in the range $[1, 10]$ excluding $[max (1, PerfLev - 2), min (PerfLev + 2, 10)]$ .

Three sets of experiments have been conducted.

Experiment#1: This experiment is conducted assuming that only RaterType#1 and RaterType#2 exist in the system. It includes nine sub-experiments with varying density of RaterType#2 from 10 to 90 percent, with a step of 10 percent.

Experiment#2: It considers the presence of only RaterType#1 and RaterType#3 in the system. It also includes nine sub-experiments with varying density of RaterType#3 from 10 to 90 percent, with a step of 10 percent.

Experiment#3: This experiment is carried out in the presence of all the three types of raters. Here, nine sub-experiments have been performed with varying combined density of RaterType#2 and RaterType#3 from 10 to 90 percent, with a step of 10 percent. In each sub-experiment, the densities of both RaterType#2 and RaterType#3 are equal.

4.2. Experimental results

For each of the sub-experiments, 10 rounds of simulations have been conducted and their average has been calculated to obtain the final reputations of the services. In all the experiments, the values for $c s_{w}$ and $c y_{w}$ are set to 0.5. Table 2 lists the RMSEs recorded in every experiment.

Table 2
RMSEs recorded by varying the density of RaterType

RaterType Density Experiment#1 Experiment#2 Experiment#3

10% 0.0636 0.0162 0.0365

20% 0.1304 0.0817 0.0838

30% 0.1977 0.2336 0.1485

40% 0.2681 0.5317 0.2353

50% 0.3393 1.0435 0.3502

60% 0.4143 1.8319 0.5057

70% 0.4916 2.8335 0.7079

80% 0.5690 3.7567 0.9798

90% 0.6528 4.3475 1.3114

RaterType Density	Experiment#1	Experiment#2	Experiment#3
10%	0.0636	0.0162	0.0365
20%	0.1304	0.0817	0.0838
30%	0.1977	0.2336	0.1485
40%	0.2681	0.5317	0.2353
50%	0.3393	1.0435	0.3502
60%	0.4143	1.8319	0.5057
70%	0.4916	2.8335	0.7079
80%	0.5690	3.7567	0.9798
90%	0.6528	4.3475	1.3114

It can be observed from Table 2 that, RMSEs recorded for any density of RaterType in both Experiment#1 and Experiment#3 are within acceptable range. It is also to be mentioned here that, in these two experiments, though the obtained RMSEs indicate slight variations in estimated reputations of services, but the actual rankings of the services are not disturbed. The same is true for density of RaterType#3 upto 60% in Experiment#2. For 70%, 80% and 90% density of RaterType#3 in Experiment#2, the obtained RMSEs are reasonably high and also the actual rankings of the services are disturbed. This is due to the presence of high percentage of malicious raters in the system. In such situations, the unfair feedback ratings submitted by these high numbers of malicious raters are granted as true and the fair feedback ratings of the honest raters are suppressed. However, Whitby et al. [24] suggest that the presence of high percentage of malicious raters in real systems is impractical and a much lower rate of malicious raters should be expected. Thus, it can be concluded that the proposed approach can fairly assess service reputations in the presence of various kinds of raters in the system.

Fig. 1.

Status of the estimated service reputations in compared to the service performance levels for Experiment#2.

Figure 1 is used to illustrate the rankings of the services for every density of malicious consumers (RaterType#3) mentioned in Experiment#2. It depicts the status of the estimated service reputations in compared to the service performance levels for Experiment#2. For each of the nine experiments, the estimated reputations of all the services that hold the same performance level are averaged to compare them against the corresponding service performance level as a whole. This is done to demonstrate the comparison in a smaller window, instead of showing the comparison for individual service’s estimated reputation with its performance level. Thus, in all the cases of Fig. 1, the ten groups of services that correspond to a distinct performance level are shown.

4.3. Performance comparison

In this section, the performance of the proposed approach has been compared with an existing reputation measurement approach from the literature. The comparison has been done under the same simulated environment setting mentioned above.

In [18], the authors proposed two service reputation assessment methods: HITSW (without the neutralization of malicious users) and HITSN (after the neutralization of malicious users). It can be observed that HITSN performs better than HITSW for any density of malicious users in the system. The performance of the proposed reputation measurement approach has been compared with HITSN in terms of RMSE.

HITSN [18] is a three-phase process for service reputation evaluation. The first phase deals with the evaluation of user credibility values which is an iterative process. Initially, considering all the users as honest (that is, all user credibility values are initialized with 1 in the first iteration), the service reputation values are evaluated using equation (18). $\begin{matrix} (18) & R_{j} = \frac{\sum_{u_{i} \in U_{j}} C_{i} * w_{i j}}{| U_{j} |} \end{matrix}$ where, $R_{j}$ denotes the reputation of service $s_{j}$ , $U_{j}$ denotes the subset of users that have submitted feedback ratings for service $s_{j}$ , $w_{i j}$ is the feedback rating submitted by user $u_{i}$ for service $s_{j}$ , $C_{i}$ is the credibility of user $u_{i}$ .

After that, the user credibility values are evaluated by equation (19). $\begin{matrix} (19) & C_{i} = \frac{\sum_{s_{j} \in S_{i}} \frac{min (w_{i, j}, R_{j})}{max (w_{i, j}, R_{j})}}{| S_{i} |} \end{matrix}$ where, $S_{i}$ denotes the subset of services that are rated by user $u_{i}$ .

The iteration continues until the reputation and credibility values are stabilized or the maximum number of iterations is reached.

In second phase, the credibility values are sorted and a threshold (θ) is calculated to identify honest and malicious users. $\begin{matrix} (20) & θ = \frac{k + l}{2} \end{matrix}$ where, k and l are the adjacent credibility values which has the largest gap.

In the last phase, the phase 1 is repeated with the refined sets of users and services.

Experiment#4: Two types of users are considered in this experiment: honest and malicious. Honest users submit their feedback ratings in the range $[max (1, PerfLev - 2), min (PerfLev + 2, 10)]$ , and malicious users submit feedback ratings outside the range. A total of 10 sub-experiments have been performed by varying the density of malicious users from 5 to 50 percent, with a step of 5 percent. The average of 10 rounds of simulations produces the final reputations of the services. The RMSEs are calculated using equation (17).

Table 3
Comparison between RMSEs of candidate methods

Malicious Raters Density HITSN Proposed Model

5% 2.2845 0.4999

10% 2.2872 0.5120

15% 2.2919 0.5361

20% 2.6771 0.5720

25% 3.1013 0.6339

30% 3.3133 0.7226

35% 3.4264 0.8507

40% 3.5385 1.0237

45% 3.6475 1.2513

50% 3.7525 1.5379

Malicious Raters Density	HITSN	Proposed Model
5%	2.2845	0.4999
10%	2.2872	0.5120
15%	2.2919	0.5361
20%	2.6771	0.5720
25%	3.1013	0.6339
30%	3.3133	0.7226
35%	3.4264	0.8507
40%	3.5385	1.0237
45%	3.6475	1.2513
50%	3.7525	1.5379

Table 3 shows the comparison between RMSEs obtained by the proposed service reputation measurement approach and HITSN. It can be observed that the proposed model outperforms HITSN, owing to the fact that the proposed model employs the measures of interrater reliability for the user credibility assessment.

5. Conclusion

Service reputation measurement is important when selecting the optimal service from a pool of functionally equivalent services. Feedback ratings are considered to be the prominent indicator of service reputation. However, because of the simultaneous presence of both malicious consumers and consumers with subjective preferences, the actual reputation of a service may be distorted. Therefore, a service reputation measurement approach is proposed in this paper accounting the diversified act of rating from various types of consumers in the system. The measure of interrater reliability is employed to devise an efficient user credibility assessment methodology. And finally, the service reputation is estimated by giving more weightage to the feedback ratings of high credible consumers and relatively less weightage to the feedback ratings of low credible consumers. Experiments are performed employing a simulated environment and results show that the proposed approach can fairly assess service reputations in the presence of various kinds of raters in the system.

References

Banković ,

J.C.

Vallejo ,

Fraga and

J.M.

Moya , Detecting bad-mouthing attacks on reputation systems using self-organizing maps, in: Computational Intelligence in Security for Information Systems,

A.´

Herrero and

Corchado , eds, Lecture Notes in Computer Science, Vol. 6694, Springer, Berlin, Heidelberg, 2011.

Booth ,

Haas ,

McCabe ,

Newcomer ,

Champion ,

Ferris and

Orchard (eds), Web Services Architecture. W3C Working Group Note, 2004, https://www.w3.org/TR/2004/NOTE-ws-arch-20040211/.

Cohen , A coefficient of agreement for nominal scales, Educational and Psychological Measurement. 20(1) (1960), 37–46. doi:10.1177/001316446002000104.

Cohen , Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit, Psychological Bulletin. 70(4) (1968), 213–220. doi:10.1037/h0026256.

Dellarocas , Immunizing online reputation reporting systems against unfair ratings and discriminatory behavior, in: Proceedings of 2nd ACM Conference on Electronic Commerce, 2000, pp. 150–157. doi:10.1145/352871.352889.

K.L.

Gwet , Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Multiple Raters, 4th edn, Advanced, Analytics, LLC, USA, 2014.

T.D.

Huynh ,

N.R.

Jennings and

N.R.

Shadbolt , Certified reputation: How an agent can trust a stranger, in: AAMAS ’06: Proceedings of the Fifth International Joint Conference on Autonomous Agents and Multiagent Systems, ACM Press, New York, NY, USA, 2006, pp. 1217–1224. doi:10.1145/1160633.1160854.

Kreger , Web Services Conceptual Architecture (WSCA 1.0). Technical Report, IBM Software Group, 2001.

J.R.

Landis and

Koch , The measurement of observer agreement for categorical data, Biometrics 33 (1977), 159–174. doi:10.2307/2529310.

10.

Malik and

Bouguettaya , Evaluating rater credibility for reputation assessment of web services, in: Web Information Systems Engineering – WISE 2007,

Benatallah ,

Casati ,

Georgakopoulos ,

Bartolini ,

Sadiq and

Godart , eds, Lecture Notes in Computer Science, Vol. 4831, Springer, Berlin, Heidelberg, 2007.

11.

Malik and

Bouguettaya , Rater credibility assessment in web services interactions, World Wide Web 12 (2009), 3–25. doi:10.1007/s11280-008-0056-y.

12.

Malik and

Bouguettaya , RATEWeb: Reputation assessment for trust establishment among web services, The VLDB Journal. 18(4) (2009), 885–911. doi:10.1007/s00778-009-0138-1.

13.

H.T.

Nguyen ,

Zhao and

Yang , A trust and reputation model based on Bayesian network for web services, in: The 8th IEEE International Conference on Web Services, 2010, pp. 251–258.

14.

T.H.

Noor ,

Q.Z.

Sheng ,

Yao ,

Dustdar and

A.H.H.

Ngu , CloudArmor: Supporting reputation-based trust management for cloud services, IEEE Transactions on Parallel and Distributed Systems. 27(2) (2016), 367–380. doi:10.1109/TPDS.2015.2408613.

15.

W.J.

Popham , Modern Educational Measurement: A Practitioner’s Perspective, Prentice Hall, Englewood Cliffs, NJ, 1990.

16.

Spearman , The proof and measurement of association between two things, The American Journal of Psychology 15(1) (1904), 72–101. doi:10.2307/1412159.

17.

Tibermacine ,

Tibermacine and

Cherif , Regression-based bootstrapping of web service reputation measurement, in: IEEE International Conference on Web Services, New York, 2015, pp. 377–384. doi:10.1109/ICWS.2015.57.

18.

Tibermacine ,

Tibermacine and

M.L.

Kerdoudi , Reputation evaluation with malicious feedback prevention using a HITS-based model, in: 2019 IEEE International Conference on Web Services (ICWS), 2019, pp. 180–187. doi:10.1109/ICWS.2019.00039.

19.

Tinsley and

D.J.

Weiss , Interrater reliability and agreement of subjective judgments, Journal of Counseling Psychology. 22(4) (1975), 358–376. doi:10.1037/h0076640.

20.

Tinsley and

D.J.

Weiss , Interrater reliability and agreement, in: Handbook of Applied Multivariate Statistics and Mathematical Modeling.,

Tinsley and

S.D.

Brown , eds, Academic Press, San Diego, CA, US, 2000, pp. 95–124. doi:10.1016/B978-012691360-6/50005-7.

21.

O.A.

Wahab ,

Bentahar ,

Otrok and

Mourad , A survey on trust and reputation models for Web services: Single, composite, and communities, Decision Support Systems. 74 (2015), 121–134. doi:10.1016/j.dss.2015.04.009.

22.

Wang ,

Zhang and

Li , A high-reliability multi-faceted reputation evaluation mechanism for online services, IEEE Transactions on Services Computing 12(6) (2019), 836–850. doi:10.1109/TSC.2016.2638812.

23.

Wang ,

Zheng ,

Wu ,

M.R.

Lyu and

Yang , Reputation measurement and malicious feedback rating prevention in web service recommendation systems, IEEE Transactions on Services Computing. 8(5) (2015), 755–767. doi:10.1109/TSC.2014.2320262.

24.

Whitby ,

Jøsang and

Indulska , Filtering out unfair ratings in Bayesian reputation systems, in: Proceedings of the Workshop on Trust in Agent Societies, at the 3rd International Joint Conference on Autonomous Agents and Multi Agent Systems (AAMAS2004), 2004.

25.

Wu ,

Yan ,

Ding ,

Liu ,

Wang ,

Jiang and

Zhou , A novel method for calculating service reputation, IEEE Transactions on Automation Science and Engineering. 10(3) (2013), 634–642. doi:10.1109/TASE.2013.2238231.

26.

Xiong and

Liu , PeerTrust: Supporting reputation-based trust for peer-to-peer electronic communities, IEEE Transactions on Knowledge and Data Engineering. 16(7) (2004), 843–857. doi:10.1109/TKDE.2004.1318566.

27.

Xu ,

Martin ,

Powley and

Zulkernine , Reputation-enhanced QoS-based Web Services Discovery, in: IEEE International Conference on Web Services (ICWS 2007), Salt Lake City, UT, 2007, pp. 249–256. doi:10.1109/ICWS.2007.152.

28.

Yang ,

Chen and

Yu , A reputation evaluation technique for Web Services, International Journal of Security and Its Applications 6(2) (2012), 329–334.

29.

Zhang ,

Kong and

Zhou , A domain partition-based trust model for unreliable clouds, IEEE Transactions on Information Forensics and Security 13(9) (2018), 2167–2178. doi:10.1109/TIFS.2018.2812166.

Applying interrater reliability measure for user credibility assessment in reputation-oriented service discovery

Abstract

Keywords

1. Introduction

2. Related work

3. Reputation measurement approach

3.1. Preliminaries

3.1.1. User credibility assessment

3.1.2. Interrater reliability

3.2. Proposed framework

3.2.1. Model setup

3.2.2. Computing consensus estimate

Table 1 Interpretations of various κ w values as suggested by Landis and Koch Value of κ w Strength of Agreement <0.00 Poor 0.00–0.20 Slight 0.21–0.40 Fair 0.41–0.60 Moderate 0.61–0.80 Substantial 0.81–1.00 Almost Perfect

3.2.4. Computing consensus and consistency estimates of single rater vs rest of raters

3.2.5. Computing user credibility

3.3. The service reputation calculation

4. Experiments and results

4.1. Experimental setup

4.2. Experimental results

Table 3 Comparison between RMSEs of candidate methods Malicious Raters Density HITSN Proposed Model 5% 2.2845 0.4999 10% 2.2872 0.5120 15% 2.2919 0.5361 20% 2.6771 0.5720 25% 3.1013 0.6339 30% 3.3133 0.7226 35% 3.4264 0.8507 40% 3.5385 1.0237 45% 3.6475 1.2513 50% 3.7525 1.5379

References

Table 1
Interpretations of various $κ_{w}$ values as suggested by Landis and Koch

Value of $κ_{w}$ Strength of Agreement

<0.00 Poor

0.00–0.20 Slight

0.21–0.40 Fair

0.41–0.60 Moderate

0.61–0.80 Substantial

0.81–1.00 Almost Perfect

Table 3
Comparison between RMSEs of candidate methods

Malicious Raters Density HITSN Proposed Model

5% 2.2845 0.4999

10% 2.2872 0.5120

15% 2.2919 0.5361

20% 2.6771 0.5720

25% 3.1013 0.6339

30% 3.3133 0.7226

35% 3.4264 0.8507

40% 3.5385 1.0237

45% 3.6475 1.2513

50% 3.7525 1.5379