Transforming agent-actions into reputational images with probabilistic planning and a fuzzy cognitive map

Abstract

This paper proposes to see the way to integrate recommendations and direct experiences into reputational images as a planning problem. Specifically we achieve this goal through a combination of a fuzzy cognitive map and probabilistic planning with RDDL. Both were implemented inside a deliberative agent architecture that intends to appropriately represent the uncertain nature of data (with a Markov decision process) and the involved deliberative reasoning (with a fuzzy cognitive map). We are interested in the viability of using this view so it was tested by the simulations inspired in the ART testbed domain, but our final intention is proposing an alternative and richer representation of concepts involved in reputation problems instead of increasing the accuracy of trust predictions at any cost.

Keywords

Trust reputation agent architectures probabilistic planning fuzzy cognitive map

1. Introduction

In recent years, the capacity offered by Multi-Agent Systems (MAS) in order to provide jointly Context-Aware services have increased. One of the reasons of these advances has been the capacity offered by agents in order to cooperate themselves in a given situational state (context). When this context involves subjective criteria or uncertainty, this cooperation makes trust playing a fundamental role to provide reliable and useful Context-Aware services. Therefore, the decision process of trusting target agents is dependent on their reputational images computed from the shared data/knowledge by the cooperation. These data have different nature/source (subjectivity, freshness, similarity, etc.), mainly third party-recommendations and previously provided services. Therefore the addressed problem consists of the process of integration of multiple data and knowledge (recommendations and past actions) representing the same real-world object (target agent) into a consistent, accurate and useful representation (reputational image or trust level).

Regarding the critical points of this trusting problem depends on the nature of source data to fuse: (i) uncertainty of the agent’s action, e.g.: one agent A does not desire to share its knowledge with the agent B; and (ii) dynamic evolution in the trust level of agents. According to these points, we suggest that considering the integration of the very different nature and source of the information and its sequential use in each iteration cycle as a probabilistic plan, and therefore trusting agents can be seen as probabilistic planners. A plan would be dynamically built at each iteration cycle, and is made of a sequence of actions such as which agents ask for reputation, for opinions, which agents to share with opinions, etc. Since the potential execution of this trusting actions depends on probabilities (concluded from reputation values) agents reasoning and acting in a trusting system become a form of probabilistic planners. Our point is that this approach might allow agents to cooperate in a efficient way, adapting themselves and responding under a trusted way to other agents.

The paper is organized as follows. Section 2 summarizes the previous contributions to this research issue. Section 3 explains ART and its terminology and protocols, discussing the reason to define a possible generalization of ART domain. Section 4 describes the hybrid architecture which has been suggested, where the first part explained the deliberative process with the MDP model and the second part, it is explained the Cognitive Module by the design that was carried out in the FCM. Finally, Section 5 explains the results of the proposed model to verify the feasibility of the deliberative and cognitive approach in MAS and finally, in Section 6 is shown conclusions and future works.

2. Related work

In recent years, trust and reputation relevance has been recognized by journals, conferences and governmental calls. The main reason is because managing both concepts is a key factor to generalize the use of software agents intelligent enough to search and select potential partners without enough prior interaction or experience [23].

Although trust and reputation are often indistinctly used, and many definitions of them have been proposed, trust stands for a more general concept that describes an attitude or predisposition to make some costly decisions based on several decision criteria such as trustee’s privacy policy, legal requirements, and system’s properties such as transparency, authenticity, confidentiality, and non-repudiation [21], reputation can be considered as just another criterion to measure the confidence level, it represents the image that an agent has shown in previous interactions according to its honesty, motivation, competence and predictability [1]. With such approach the reputation of an agent can be seen as a statistical value about the trust probability computed from previous interactions and recommendations. In this way, reputation-based trust models should provide an incentive mechanism that would decrease the level of risks involved in the interaction with potential malicious agents [9].

Many reputation-based trust models have been published as these surveys show [29,31], but from them, we can distinguish two main alternatives: The ones exclusively based on mathematical methods such as Bayesian probability, maximum likelihood, Beta probability, weighted arithmetic means, game theory, and average of weighted recommendations [4,15,18,22,30,35,38,40], while others pay attention to broader requirements which these models have to request [2]: (i) cognitive approach to resolve complex trust problems; (ii) reasoning of the agent as a module of the multi-agent architecture, providing agents of reactive capacity; and (iii) interoperable solution to compare different trust models. These belief/cognitive trust models are built on abstractions of the human concept of trust [11,39].

Another relevant issue is determining which of this trust model is more effective. However, performance of trust models tends to be evaluated with ad-hoc simulations and metrics that combine different frequency and intensity of change in agent behaviours. Therefore it was difficult to conclude fair comparisons among the so many proposed reputation-based trust models, and the Agent Reputation Testbed (ART) initiative was launch with this aim [14].1

¹
http://megatron.iiia.csic.es/art-testbed.

Three international competitions took place jointly with AAMAS international Conferences and the agents implemented for them produced several publications (sabatini [16], iam [36], afras [5], simplet [25], peles [10] and uno2008 [28]). The main criticism of ART testbed was the lack of a proper scalability enough to motivate the central role that reputation should play in trusting decisions.

The claim of this work is to present this agent trust reasoning process following a cognitive approach guided by a probabilistic planning as the Markov Decision Process (MDP). Although, as we have mentioned some of them before, the use of probabilistic approach to trust models is not new, these approaches do not combine it with a deliberative reasoning architecture. Furthermore, the originality of this paper also comes from the point that no author has seen this trust reasoning process from the planning perspective as we suggest to do. In both senses our contribution is innovative.

The closest contributions to our proposed architecture that adds a socio-cognitive module into the agent’s trust strategy, are the theories proposed in [7,12] with Fuzzy Cognitive Maps (FCM) and the integration of a socio-cognitive module with ART architecture in [20]. Another obvious precedent is the first approach in the design of the trusting process as planning problem was carried out in [6] using Probabilistic Planning Domain Definition Language (PPDDL). However, this first attempt has shown a number of critical points which PPDDL has not managed correctly. It can be summarized in: (i) preconditions with probabilities; (ii) definition of dynamic rewards (in discrete time); and (iii) usage of non-quantitative probabilities.

3. ART terminology and protocols

Agent Reputation Testbed (ART), defined in [14], is a multi-agent platform to compare different reputation models in the appraisal domain. In this domain, agents are players/competitors that appraise paintings and implement trust strategies. Very close valuations of paintings to the real value would lead to more future clients, and therefore to more earning so win the competition. Each painting belongs to an era among a finite set of possible artistic eras. All the interactions which take place in the ART domain can be enumerated in the following steps:

Client makes a request to appraiser agent to ask the evaluation of a painting over an era.

Appraiser tries to make the evaluation but if he does not have the enough knowledge to carry out the appraisal, he will have to ask to other appraisers to share the knowledge over the requested era or share its own evaluation for the same painting and era. This task is named Reputation process.

This appraiser decides to reply or not according to its strategy in the process of sharing the reputation knowledge. In consequence, agents can perform different strategies to obtain a better result in the competition: (i) getting more money if they share the reputation; or (ii) generating bad opinions by appraisers and so they are penalized by clients.

The aim of ART platform is to provide a ranking of best agents in terms of efficiency (economic balance results) obtained from the sum of the received rewards from their actions: performing correctly opinions to clients over the pair painting – era and selling its knowledge or sharing opinions with other appraisers. While the way to obtain such rewards is choosing the best partner in each interaction in terms of reputational image obtained from the perception of its actions. This research does not aim to increase ART capabilities, the utility of ART testbed for us is because it has clearly specified the steps to be taken in each trusting iteration cycle while all concepts involved are clearly stated and defined. So we can use ART testbed as a case of use of a probabilistic planner acting as a trusting agent.

Regarding the previous description, ART Testbed was defined to the appraisal domain, but to generalize our approach we renamed concepts and predicates to achieve the independence of the domain in the model. In this sense, we propose: Service instead of Painting; Type of Service instead of Artistic Era; Provider Agent instead of Appraiser Agent; and Capacity instead of Certainty.

4. Trusting process as a probabilistic planning problem

We decided the usage of Relational Dynamic Influence Diagram Language (RDDL), developed by Sanner in [32] because RDDL could model complex probabilistic domains more effectively due to the expressiveness which is introduced in the following points:

Usage of fluents variables with in the definition of every domain element: actions, states, constants and observations;

Support to define dynamic strategies for rewards;

Possibility of using non-quantitative probabilities;

Support to use the most important probability distributions: KronDelta, DiracDelta or Bernoulli;

Management of the concurrence in action preconditions/effects. Due to the design by a Dynamic Bayesian Network (DBN) of RDDL, it is not allowed conflict in the global restrictions of the domain because action preconditions are not checked locally.

Additionally RDDL is becoming the standard language where probabilistic planning Community is going to focus their efforts in searching new designs and improvements; for instance, International Probabilistic Planning Competition (IPPC) 2011.2

²
Usage of RDDL in IPPC 2011, see http://users.cecs.anu.edu.au/~ssanner/IPPC_2011.

We have also another reason to choose RDDL: it puts forward a design which was shaped taking the flexibility as a significant point. This fact has offered the possibility of mixing the two kinds of layers to model the trust process for the agent: Deliberative and Cognitive layers. However, this solution is not valid to be integrated in the official ART Simulator (there would be to implement a middleware between this architecture and ART simulation engine and we did not take such approach in this paper).

Since the aim of this work is to propose a new deliberative approach in the MAS communication exploring solutions in Cognitive Trust [13,20,37], the proposed agent architecture embraces: (i) a Deliberative Layer, which is in charge of solving the communication problem through a DBN [3,32]; and (ii) a Cognitive Layer, which computes the fusion of heterogeneous data into the concept of trust in this problem in order to supervise the communication of the agent in a reliable way [7,12]. This architecture lead the agent to create an action plan that agent will use it to infer what type of message (opinion or reputation) has to be sent to providers in each step of the ART simulation.

Fig. 1.

Hybrid architecture for a trusted communication in ART agent.

Figure 1 shows the main components of the agent architecture. The client updates the set of Variables, that is to say a new request predicate is put as a initial state for the RDDL planner. Then, the planner starts to instantiate the agent-action from the Action Base maximizing the Reward function (to get the highest number of coins in ART Simulator) according to Constraints (of the agent-action, i.e.: available coins, necessity of knowledge, etc.) and the trust-level of each Provider-Agent has got from the FCM in the Cognitive Module of the agent. Therefore, every agent-action is required to obtain a minimum trust-level during the decision process in order to be instantiated.

4.1. Definition of the probabilistic planning domain in RDDL

RDDL uses the Markov Decision Process (MDP) to model problems with sequential decision where the system is evolving in the time and it is controlled by an agent [3,33], assuming this agent will always know its own state before applying their actions. A MDP is described formally as the following variable $M = ⟨ S, A, Φ, R ⟩$ , where S and A are the finite sets of states and actions, respectively; $Φ : A \times S \to Π (S)$ is the transition function, denoted by the relationship $Pr (s, a, t)$ which specifies the probability of select the state x being the system in the state s, applying the action a in the time t; finally, $R : A \times S \to ℜ$ is the reward-function which the system receives after putting to use the action a in the state s.

According to this, it is defined the stationary policy as the association $π : S \to A$ that chooses an action a for the state s to specify the agent’s plan. Assuming the agent runs a finite steps, named finite horizon, the value of a policy is given by $V_{n}^{π} (s) \to ℜ$ to evaluate the optimal value of the policy π in n steps from the state s as: $\begin{array}{rclr} V_{n}^{π} (s) & = & R (s, π (s)) \\ + γ \sum_{s^{'} \in S} Φ (π (s), s, s^{'}) V_{n - 1}^{π} (s^{'}), & (1) \end{array}$ where γ is the discounted factor defined in $0 ⩽ γ < 1$ and it is employed to discount rewards over future actions given the uncertainty which rules the process.

The main problem which MDP planners have to resolve is in the resolution of problems where the size and dimensionality are too high. According to this, it has been described in [8,17] that one of the two critical points that MDP planners have to manage is the exponential growth of DBN model when the number of random variables increase.

Then, the section is focused in formalizing the types, predicates and actions (asking-opinion and asking-reputation) under the probabilistic planning paradigm. The basic concept of ART domain are: (i) agent, both clients and appraisers; (ii) service; and (iii) type of service. Previously, it is defined the RDDL requirements, following the same process like other languages with imports. In this case, the domain is provided to manage real and integer numbers, deterministic reward function, intermediate states and the usage of global constraints for states (see Fig. 2).

Fig. 2.

RDDL Basic concepts in ART domain.

Fig. 3.

Examples of state-variables according to the MDP model proposed by RDDL.

Fig. 4.

Transition map for the Fluent variable has-reputation.

Then, it is described the variables which will model the different states where the previous concepts can be. There are the following groups to manage these variables:

Non-fluents variables, those variables will be remained constant during all planning process; it will only change in the definition of each problem. They represent the different constants of the domain and the predicates whose value does not change along time.

Fluents variables, they are the states in the DBN model. They are those variables whose value changes along time–execution, either stochastic way as is-cooperative(agent) variable or determined by agent-action/other variables (both fluents or non-fluents).

Agent-Actions, they represent the actions in the PPDDL model and they enclose the two main actions that an agent carries out: asking-reputation and asking-opinion. Regarding that: (i) action preconditions are defined in the point Action Constraints in this section; and (ii) action effects are performed in the transition model cfp, specified latter.

In the following lines, it is presented an example of RDDL for each one of the previous classes of variables. Regard that the whole definition of RDDL variables is defined in the Appendix of this document (see Fig. 3).

Fig. 5.

Example of transition map for thresholds-variables.

Fig. 6.

Reward function defined for each ART agent.

Fig. 7.

Preconditions in agent-action asking-reputation for global constraint in MDP model.

In the next point, it is defined the logic in the MDP model, named in RDDL as Conditional Probabilistic Function (cfp), that is to say transitions which Fluent variables has to carry out.

time, it is a deterministic variable to model the discrete time.

is-cooperative, it is random variable according to the Bernoulli distribution and so the agent’s behaviour to cooperate with others (during reputation process) is guided by this variable.

has-capacity bank-balance, they represent the Agent’s Knowledge in a concrete Type of Service and the money which agent has earned, respectively. It can be said these variables are indirect random variables because they are dependent on the random variable is-cooperative over asking-reputation and asking-opinion actions.

has-feedback, is-friendship and is-lier, they are the states where it is offered to the Agent the information about the Direct Experience which he (the Agent) has perceived in the past about a Provider.

Below and more specifically, it is shown the Fluent has-reputation in RDDL language as one of the most representative variable in the ART domain. As we say previously, regard that the rest of the transition map is allocated in the Appendix of this document (see Fig. 4).

According to [6, Section 6], it has to be designed an additional strategy over the previous logic. Agents does not know a priori the behaviour of others in their process where they exchange and share the reputation.

In this sense, it is set out a exploration vs. exploitation strategy (similar to others agent competitions) where the agent may have the possibility to acquire the knowledge which others agents have got in the first steps of the planning process. Along this planning process draws on, agents will change toward a exploitation of their knowledge instead of their initial behaviour of exploration.

To achieve this, it has been employed a technique based on thresholds which rule the trigger of agents’ actions according to the values obtained in the knowledge, capacity, cooperation and experience threshold-variables. Below, it is shown an example to control the cooperation variable (see Fig. 5).

Fig. 8.

Fuzzy cognitive map to calculate the trust-level for each Provider Agent in actions.

These thresholds are defined ad-hoc, as they were defined in the other implemented and published ART competition agents. The chosen values have impact on the accuracy of trust predictions, and the specific instantiation of them could be relevant to win ART games against published competition agents, but in fact one of the conclusions of the ART discussion notes3

http://megatron.iiia.csic.es/art-testbed/pdf/PostCompetitionDiscussion.pdf.

hold in AAMAS conferences was the need to avoid such over specialization.

One of the most important point in the design of probabilistic planning is the definition of the reward function. In the ART context, the designed function compensates to agents due to the cooperation which may be exchanged among them, either in the reputation or opinion processes.

Regarding that agents must search a trade-off solution between capacity over types of services and invested money in getting the knowledge to perform good opinions, i.e. REPUTATION-COST and OPINION-COST when fluent-state bank-balance is calculated (see Fig. 6).

We used the First-order logic in order to specify preconditions, inhibiting the available actions for each agent when any precondition is not asserted. Taking into the logic for the Fluent variable has-reputation as the example to define, the asking-reputation action must be satisfied the following rule (see Fig. 7).

All it is showed up to this moment is the Probabilistic planner following MDP model to create a plan of action for the communication among agents. However, the reader may have checked that the main state related about the trust-level has not defined yet; i.e.: has-trust. In the next section, it is explained the process where the trust-level is worked out for each Provider through a fuzzy cognitive map and considering the others states (of the MDP model) which take part in the inference-chain.

4.2. Inference of the trust-level using FCM

The fluent state has-trust has been realized through a Fuzzy Cognitive Map (FCM) to model the system [24,34]. This graph with feedback consists of nodes, which model concepts, and their interconnections, which represent causal relations, weighted to identify what role plays each interconnection. At each computation step, the value of every concept is calculated, computing the influence of other concepts to this concept, by applying the following equation: $x_{i} (t) = f (\sum_{j = 1, j \neq i}^{n} x_{j} (t - 1) w_{j i}),$ (2) where $x_{i} (t)$ is the value of the concept i at time t; $w_{i j}$ is the weight of the interconnection between the concept i and j; and f is the activation function that may be an Identity, Unit, Sigmoid or Gaussian function.

FCM integrates the accumulated experience, making it available a straightforward decision in different applications and domains where the knowledge of human expert can be model.

Figure 8 shows the FCM proposed to calculate what trust-level has the provider agent over the agent may instantiate the asking-reputation and asking-opinion actions. As well as [13,37], it has been displayed following a tree-like structure having the Trust as root concept, therefore, the agent will calculate the Trust concept according to Eq. (2) to retrieve the final trust-level for this provider agent. Then, it is connected the three factors which categorize Trust in the context of this problem and they are going to explain as follows:

Direct Experience, it manages the past experience that was got from that provider, taking into account whether the provider had a possible friendship-relation, as well as UNO strategy defined in [27].

Reputation Experience, it defines the trust information about this Provider by others, regarding that reputation information is weighted according to the direct experience that the agent had about others which make this reputation; see the interconnection between direct experience node and the provider agent node to model this kind of feedback.

External Factors, it concerns the opportunity of performing the ART Coordination protocol in order to increase the own capacity, so it has been defined the necessity by: 1 − has-capacity.

This application of FCM on Trust problems allows to prune those actions that the fluent-state: has-trust(?a, ?p), where p is the provider agent of the action, is lower than the THRESHOLD_TRUST variable. ART Agents can thus resolve the coordination as a planning problem by guiding their actions according to a Trust-Cognitive module.

Table 1

Percentages of wins by the ART agent with FCM over fully deliberative agents

Problem	No. agents	Honest agents	Unfair agents	FCM agent’s wins	Time (s)
1	4	1	1	60%	31.87
2	7	2	3	70%	103.75
3	9	3	4	75%	250.68
4	11	4	5	85%	450.17
5	13	5	6	90%	897.41

5. Evaluation

In this section, the proposed architecture was built to demonstrate the performance of the cognitive model under two experimental scenarios. The purpose is to evaluate the behaviour of the trust strategy, the interaction and cooperation with other agents and how to manage unfair agents.

5.1. Evaluation settings

The evaluation was analysed following two different approaches. All experiments were implemented using SPUDD,4

⁴
General information about SPUDD and source-code download, see http://www.cs.uwaterloo.ca/~jhoey/research/spudd/index.php.

a Probabilistic planner proposed in [19]. In a concise way, SPUDD tries resolving Eq. (1) to MDP problems iterating in values until

V_{n}

, where n is the defined value in the horizon parameter in the equation and using the Algebraic Decision Diagrams (ADDs). ADDs are generalizations of ordered Binary Decision Diagrams (BDDs) that allow the search in functions like

B^{n} \to ℜ

for n boolean variables. In the MDP context, ADDs is the technique to explore the search space defined by the functions:

R : A \times S \to ℜ

and

V_{i + 1}^{π} (s_{i}) \to ℜ

, and so the maximum values will be reached in diagram branches. In our experiments, we consider an horizon value

h = 20

and

γ = 0.9

to be applied in the SPUDD planner taking these values as reference in [32].

Table 2

Results in simulation process where an agent can instantiate actions concurrently in MDP problems by means of RDDL

Problem	Concurrent actions	FCM agent’s wins (%)	Time (s)

			FCM agent	Deliberative agent
1	1	93%	30	31
2	2	96%	1500	1680
3	4	100%	16,600	18,000

Concerning about the set-up of FCM, experiments have to define two values to initialize the map. First, the weight of each link in the FCM, shown in Fig. 8, reflects the impact of the corresponding concept in the final trust. In this work, these weights are constant and this FCM does not utilize any learning technique to update them. In consequence, we have decided to set out the FCM with 50% for Direct Experience, 40% for Reputation Experience and 10% in External Factor. Second and about the activation function, we have decided to use the Sigmoid Function $f (x) = \frac{1}{1 + e^{- x}}$ in order to normalize trust-level values inside the interval $(0, 1)$ , so there is no negative trust level values although an agent would be dishonestly acting.

5.2. Evaluation of the ART agent with FCM against fully deliberative agents

In the first experiment, the aim was to compare two kind of agents. The first one was defined by the FCM template of this work in the process of instantiation of agent-actions. The other kind of agents defines the rest of agents implemented by the initial version of RDDL in ART problem [26], where the inference process were managed by the deliberative planner SPUDD. Besides, for each problem it is added cheating providers, which does not collaborate, and honest providers, which always performs the best opinion and reputation for requesters.

The results are showed in the Table 1. As it may be appreciated, strategies which were guided by the FCM have won in all problems (better than 50% of wins). It is also important to note that the percentage of wins for our FCM agent increases according to the number of agents does in each problem, both honest and liars agents. For this reason, a trust-cognitive strategy is more optimal in coordination problems where relationships among agents are increased and it is needed some mechanism to control the fusion process.

When the problem has set up with only 4 agents, Cognitive agent does not improve significantly instead of the Deliberative agent; SPUDD planner got a 50% of success to instantiate the agent-action. However, when the number of agents is increased, agent’s reasoning with FCM offers an efficient results, both Agent’s Wins and Time Reduction, due to:

Management about providers (Who is a liar?, Did I receive a good feedback from that provider?, etc.) by Direct Experience.

Making trust-bonds with providers through the Reputational Experience.

Evaluation whether new opinions should be gathered from providers by means of Knowledge Necessity.

5.3. Evaluation in problems with concurrent agent-actions

In the second experiment, the goal was to identify the expected improve when our agent is deployed in a more realistic environment by means of allowing a higher number of parallel actions for each agent, that is to say a concurrent MDP problem. All problems have been set up with the identical parameters: 5 services, 3 types of service and the same features in SPUDD. Regard that the number of agents has been set up in four agents for all problems: our FCM agent, one non-cognitive, a cheating/liar provider and finally, an honest provider; RDDL cannot manage a higher number of agent due to a scalability problem when agent-actions are performed concurrently.

The experiment, presented in the Table 2, showed a satisfactory results. On the one hand, results haven confirmed that our trust-cognitive strategy offers a better response according to the environment is more realistic. Our FCM agent got better result for problems with four agents where it is allowed to use parallel agent-actions, as well as the number of agents increased in the previous evaluation.

On the other, there is a loss of the scalability when concurrent agent-actions grows. We can only launch three problems under this context, however, FCM agent gives a significant reduction in the execution time in contrast with results offered by the initial deliberative agent in [26].

6. Conclusions and future works

Addressing the coordination process in Multi-Agent Systems as a probabilistic planning problem of Deliberative agents offers a new, descriptive and complete solution to trust and reputation discipline. We introduce a Deliberative Architecture based on the concept of Trust proposed by Castelfranchi and Falcone and after this work, we have confirmed the possibility of integrating a deliberative reasoning guided by Cognitive Trust module in a seamless way, thanks for the flexibility which RDDL offers. In the first evaluation, our agent guided by fuzzy cognitive map decides a better plan for the ART coordination problems thanks to a better state-space search as a result of: (i) efficient modelling for Trust and Reputation concept to detect the nature of agents (reliable vs. liar; trust vs. mistrust providers) and, (ii) agent awareness to represent its knowledge level (either innate or gathered from other agents) and the necessity to improve it.

In second evaluation about concurrent MDP problems, FCM agent does not offer the expected result due to the loss of scalability according to the parallel actions are incremented. In a full deliberative approach we see the same problem [26], so RDDL planner as well as the hardware environment may be the reasons to explain this disadvantage. Despite this lack of scalability, we present an innovative approach that addresses the deliberative reasoning for trusting decision as a probabilistic planning Problem improved by a fuzzy cognitive map to make more reliable the trust predictions.

Future works will be discoursed in two lines: (i) we will give a learning machine for the fuzzy cognitive map to complete the design, for example we could update weights in the interconnections in consonance with the feedback protocol in ART either perform new strategies to balance the weight the reputation or opinion actions in the value of the Trust concept; and (ii) we will have to plan to translate this agent architecture in a specified Programming language such as JADE or JASON to propose an evaluation of this work in more complex scenarios than RDDL allows, including comparisons with other Trust Cognitive strategies for Multi-Agent Systems such those who participate in ART competitions.

Footnotes

Acknowledgements

This work was supported in part by Projects MINECO TEC2012-37832-C02-01, CICYT TEC2011-28626-C02-02, CAM CONTEXTS (S2009/TIC-1485).

Implementation of the probabilistic planning domain in RDDL

This appendix presents the complete design of the probabilistic planning domain in RDDL. It has been divided in the different sections which defined a domain in RDDL, i.e.: first, definition of state-variables; second, transition map for each state-variable including the variable has-trust which rules the socio-cognitive level for the agent; third, it is shown the reward function for each agent again; finally, it is presented the mechanism to restrict the application of agent-actions by means of thresholds variables and preconditions.

References

[1]

Becerra,

Heard,

Kremer,

Denzinger and

Systems, Trust attributes, methods, and uses, in: Workshop on Trust in Agent Societies, AAMAS-2007, Honolulu, HI, USA, 15 May 2007, 2007, pp. 1–6.

[2]

Bellifemine,

Caire and

Greenwood, Developing Multi-Agent System with JADE, Wiley Series in Agent Technology, 2007.

[3]

Boutilier, Symbolic dynamic programming for first-order MDPs, in: IJCAI, Morgan Kaufmann, 2001, pp. 690–700.

[4]

Buchegger and

J.-Y.L.

Boudec, A robust reputation system for P2P and mobile ad-hoc networks, in: Proceedings of P2PEcon, 2004.

[5]

Carbo and

J.M.

Molina, An extension of a fuzzy reputation agent trust model (afras) in the art testbed, Soft Computing 14(8) (2010), 821–831.

[6]

Carbo and

J.M.

Molina, Modeling art as a planning problem, in: Proceedings of the 13th International Workshop on Trust in Agent Societies (TRUST10), 2010.

[7]

Castelfranchi and

Falcone, Principles of trust for MAS: Cognitive anatomy, social importance, and quantification, in: Proceedings of the 3rd International Conference on Multi Agent Systems, ICMAS’98, IEEE Computer Society, Washington, DC, USA, 1998, pp. 72–79.

[8]

Ma.

de Guadalupe García-Hernández,

Ruiz-Pinales,

Onaindia,

J.G.

Aviña-Cervantes,

S.E.

Ledesma-Orozco,

Alvarado-Mendez and

Reyes-Ballesteros, New prioritized value iteration for Markov decision processes, Artificial Intelligence Review 37(2) (2012), 157–167.

[9]

Dellarocas, The digitization of word of mouth: Promise and challenges of online feedback mechanisms, Management Science 49(10) (2003), 1407–1424.

10.

[10]

Diniz Da Costa,

C.J.

Lucena,

Torres Da Silva,

S.C.

Azevedo and

F.A.

Soares, ART competition: Agent designs to handle negotiation challenges, in: Trust in Agent Societies, Springer, Heidelberg, 2008, pp. 244–272.

11.

[11]

Esfandiari and

Chandrasekharan, On how agents make friends: Mechanisms for trust acquisition, in: Proceedings of the Fourth Workshop on Deception, Fraud and Trust in Agent Societies, Montreal, Canada, 2001, pp. 27–34.

12.

[12]

Falcone and

Castelfranchi, Social trust: A cognitive approach, in: Trust and Deception in Virtual Societies,

Castelfranchi and

Y.-H.

Tan, eds, Springer, The Netherlands, 2001, pp. 55–90.

13.

[13]

Falcone,

Pezzulo and

Castelfranchi, A fuzzy approach to a belief-based trust computation, in: Trust, Reputation, and Security: Theories and Practice, Lecture Notes on Artificial Intelligence, Springer, 2003, pp. 73–86.

14.

[14]

Fullam,

Klos,

Muller,

Sabater,

Schlosser,

Topol,

K.S.

Barber,

Rosenschein,

Vercouter and

Voss, A specification of the Agent Reputation and Trust (ART) testbed: Experimentation and competition for trust in agent societies, in: The Fourth International Joint Conference on Autonomous Agents and Multiagent Systems, 2005, pp. 512–518.

15.

[15]

Gambetta, Can we trust trust? in: Trust: Making and Breaking Cooperative Relations, Basil Blackwell, 1988, pp. 213–237.

16.

[16]

Gomez,

Carbo and

Benac, Honesty and trust revisited: The advantages of being neutral about other’s cognitive models, Journal Autonomous Agents and Multi-Agent Systems (JAAMAS) 15(3) (2007), 313–335.

17.

[17]

Guestrin,

Hauskrecht and

Kveton, Solving factored MDPs with continuous and discrete variables, in: Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, AUAI Press, 2004, pp. 235–242.

18.

[18]

Haghpanah and

Desjardins, PRep: A probabilistic reputation model for biased societies, in: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012), 2012.

19.

[19]

Hoey,

St-Aubin,

Hu and

Boutilier, SPUDD: Stochastic planning using decision diagrams, in: Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence, 1999, pp. 279–288.

20.

[20]

J.F.

Hübner,

Lorini,

Herzig and

Vercouter, From cognitive trust theories to computational trust, in: Proceedings of the 12th International Workshop on Trust in Agent Societies, Vol. 10, Budapest, Hungary, 5 October–5 November 2009, 2009.

21.

[21]

A.R.C.

Hussin,

L.A.

Macaulay and

Keeling, in: Proceedings of the 11th Pacific Asia Conference on Information Systems, PACIS 07, Auckland, New Zealand, 3–7 July 2007, 2007.

22.

[22]

Josang and

Haller, Dirichlet reputation systems, in: 2012 Seventh International Conference on Availability, Reliability and Security, 2007, pp. 112–119.

23.

[23]

Josang,

Ismail and

Boyd, A survey of trust and reputation systems for online service provision, Decision Support Systems 43(2) (2007), 618–644.

24.

[24]

Kosko, Fuzzy cognitive maps, International Journal Man-Machine Studies 24 (1986), 65–75.

25.

[25]

J.F.H.Y.

Krupa and

Vercouter, Extending the comparison efficiency of the ART testbed, in: Proceedings of the First International Conference on Reputation: Theory and Technology – ICORE 09,

Paolucci, ed., Gargonza, Italy, 2009.

26.

[26]

J.M.

Moya and

Carbo, Applying RDDL in ART domain for multi-agent systems, in: Intelligent Systems for Context-Based Information Fusion, November 2012, 2012.

27.

[27]

Munoz and

Murillo, Agent UNO: Winner in the 2nd Spanish ART competition, Revista Iberoamericana de Inteligencia Artificial 12(39) (2003), 19–27.

28.

[28]

Munoz,

Murillo,

Lopez and

Busquets, Strategies for exploiting trust models in competitive multi-agent systems, in: Multiagent System Technologies,

Braubach,

van der Hoek,

Petta and

Pokahr, eds, Lecture Notes in Computer Science, Vol. 5774, Springer, Heidelberg, 2009, pp. 79–90.

29.

[29]

S.D.

Ramchurn,

Huynh and

N.R.

Jennings, Trust in multi-agent systems, Knowledge Engineering Review 19(1) (2004), 1–25.

30.

[30]

Rosaci, Integrating trust measures in multiagent systems, International Journal of Intelligent Systems 27(1) (2012), 1–15.

31.

[31]

Sabater and

Sierra, Review on computational trust and reputation models, Artificial Intelligence Review 24(1) (2005), 33–60.

32.

[32]

Sanner, Relational Dynamic Influence Diagram language (RDDL): Language description, NICTA and the Australian National University, 2010.

33.

[33]

Sanner and

Boutilier, Approximate solution techniques for factored first-order MDPs, in: ICAPS-07, 2007, pp. 288–295.

34.

[34]

C.D.

Stylios,

V.C.

Georgopoulos and

P.P.

Groumpos, The use of fuzzy cognitive maps in modeling systems, in: Proceedings of the 5th IEEE Mediterranean Conference on Control and Systems, Paphos, 1997.

35.

[35]

Tavakolifard and

S.J.

Knapskog, A probabilistic reputation algorithm for decentralized multi-agent environments, Electronic Notes in Theoretical Computer Science 244 (2009), 139–149. Proceedings of the 4th International Workshop on Security and Trust Management (STM 2008) .

36.

[36]

W.T.L.

Teacy,

Huynh,

Dash,

Jennings,

Patel and

Luck, The ART of IAM: The winning strategy for the 2006 competition, in: Proceedings of Trust in Agent Societies WS, AAMAS 2007, 2007.

37.

[37]

Venanzi,

Piunti,

Falcone and

Castelfranchi, Reasoning with categories for trusting strangers: A cognitive architecture, in: 14th International Workshop on Trust in Agent Societies at AAMAS 2011, February 2011, 2011.

38.

[38]

Vogiatzis,

MacGillivray and

Chli, A probabilistic model for trust and reputation, in: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2010), Morgan Kaufmann, 2010, pp. 225–232.

39.

[39]

Wang and

M.P.

Singh, Trust representation and aggregation in a distributed agent system, in: International Conference on Artificial Intelligence (AAAI), AAAI Press, Boston, MA, USA, 2006, pp. 1425–1430.

40.

[40]

Whitby,

Jsang and

Indulska, Filtering out unfair ratings in Bayesian reputation systems, in: International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), New York, US, 2004.

Transforming agent-actions into reputational images with probabilistic planning and a fuzzy cognitive map

Abstract

Keywords

1. Introduction

2. Related work

1 http://megatron.iiia.csic.es/art-testbed.

4. Trusting process as a probabilistic planning problem

2 Usage of RDDL in IPPC 2011, see http://users.cecs.anu.edu.au/~ssanner/IPPC_2011.

5.1. Evaluation settings

4 General information about SPUDD and source-code download, see http://www.cs.uwaterloo.ca/~jhoey/research/spudd/index.php.

5.3. Evaluation in problems with concurrent agent-actions

6. Conclusions and future works

Footnotes

Acknowledgements

Implementation of the probabilistic planning domain in RDDL

References

¹
http://megatron.iiia.csic.es/art-testbed.

²
Usage of RDDL in IPPC 2011, see http://users.cecs.anu.edu.au/~ssanner/IPPC_2011.

⁴
General information about SPUDD and source-code download, see http://www.cs.uwaterloo.ca/~jhoey/research/spudd/index.php.