Human-robot interaction through adjustable social autonomy

Abstract

Autonomy is crucial in cooperation. The complexity of HRI scenarios requires autonomous robots able to exploit their superhuman computations (based on DNN, Machine Learning techniques and Big Data) in a trustworthy way. Trustworthiness is not only a matter of accuracy, privacy or security, but it is becoming more and more a matter of adaptation to humans agency. As claimed by Falcone and Castelfranchi, autonomy means the possibility of dislaying or providing an unexpected behavior (including refusal) that departs from a requested (agreed upon or not) behavior. In this sense, the autonomy to decide how to adopt a task delegated by the user, with respect to her/his own real needs and goals, distinguishes intelligent and trustworthy robots from highly performing robots. This kind of smart help can be provided only by cognitive robots able to represent and ascribe mental states (beliefs, goals, intentions, desires etc.) to their interlocutors. The mental states attribution can be the result of complex reasoning mechanisms or can be fast and automatic, based on scripts, roles, categories or stereotypes typically exploited by humans every time they interact in everyday life. In all these cases, robots that build and use cognitive models of humans (that have a Theory of Mind of their interlocutors), have to operate also a meta-evaluation of their own predictive skills to build those models. Robots have to be endowed with the capability to self-trust their skills to interpret the interlocutors and the context, for producing smart and effective decisions towards humans.

After exploring the main concepts that make collaboration between humans and robots trustworthy and effective, we present the first of a series of experiments draw for testing different aspects of a designed cognitive architecture for trustworthy HRI. This architecture, based on consolidated theoretical principles (theory of social adjustable autonomy, theory of mind, theory of trust) has the main goal to build cognitive robots that provide smart, trustworthy collaboration, every time a human requires their help. In particular, the experiment has been designed in order to demonstrate how the robot’s capability to learn its own level of self-trust on its predictive abilities in perceiving the user and building a model of her/him, allows it to establish a trustworthy collaboration and to maintain a high level of user’s satisfaction, with respect to the robot’s performance, also when these abilities progressively degrade.

Keywords

Trustworthy HRI social autonomy adaptation theory of mind cognitive modelling

1 Introduction

In today’s world, artificial intelligence systems are playing a crucial role in our daily lives. The decisions made by machines are leaving a profound impact on our society and are involving almost every aspect of our life. Different kinds of artificial systems, whose behaviours are based on statistical tools, AI algorithms, machine learning models are used in applications such as healthcare, government, business, judicial and political spheres. Decisions made by AI systems lead to beat some of the best human player [26], to make super accurate medical diagnostics [13], to help companies in customers support [16] and so on. These decisions are more oriented to superhuman computations and performances, than brain-inspired or psychological paradigms. With the enormous impact that AI systems have on society, it is crucial to ensure that all these systems we are relying on are trustworthy. Trustworthy AI is largely considered one of the topics much more demanding in the artificial intelligence field, not only in research, but also in institutions [8, 27], due to the huge impact that AI systems are having in society.

As mentioned above, AI moved from human psychology inspired models (i.e. decision trees in expert systems) to deep neural networks, machine learning, Big Data and so on. If this type of approach proved to be very powerful in computational and performance terms, it increased the gap between super intelligent agents and humans, in terms of trustworthy cooperation between humans and artificial systems. We do not consider just the cases in which results provided by artificial systems have been extremely dangerous for humans [20, 25] (trustworthiness as accuracy, robustness, privacy, security etc.); we focus on those dimensions of trustworthiness (e.g. adaptation to human autonomy) that are involved when humans and artificial systems, in particular robots [11], have to interact [9, 10] and cooperate [7] with each other, and humans have to establish a deep relationship of trust [18 , 22] every time they have the necessity to include robots as part of their plans or goals (task delegation and adoption [5]). Trust is not just the result of the frequency with which an agent produces the desired behavior or result; trust is a much more complex attitude, including a causal attribution, an estimation, an ascription of several internal factors that play a causal role in the activation and control of the behavior; trust is the counterpart of delegation [6].

In this work we propose a preliminary, simulated experiment, designed with the goal to test a cognitive architecture [4] for trustworthy human robot collaboration. The architecture allows a BDI [24] robot to integrate different cognitive skills, with the goal to provide a smart help every time a human user delegates to it a task to achieve in her/his place.

2 Cooperation and task delegation

Cooperation is based on different and complementary kinds of attitudes and reasons from the partners involved. Let’s consider the following collaborative scenario (Fig. 1): a human X (the trustor) and a robot Y (the trustee) collaborate so that X has to trust Y, in a specific context, for executing a task τ (referred to an action (α), to its resulting world state (g), or both) and realizing the results that include or correspond to the X’s Goal_X (g) = g_X [6].

Fig. 1

Collaborative scenario based on delegation and adoption attitudes.

In this context, X relies on Y for realizing some part of the task she/he has in mind (task delegation); on its side, Y decides to help X, to replace her/him and perform a sequence of actions that are included in the X’s plan, in order to achieve some of her/his goals or sub-goals (task adoption). The capability to implement a smart task adoption distinguishes a collaborator from a simple tool, and presupposes intelligence and autonomy [12]. Being truly cooperative implies more than the simple concept of execution of a prescribed action. For example, in order to adopt some goal of X in an intelligent form, Y has to understand the X’s mental states (i.e. goals, beliefs, expectations about Y’s behavior) and it has to adjust the delegated action to the represented mental states, to the context and to its own current abilities and characteristics. In their much complex sense, cooperation and help require more autonomy and initiative. A real collaborative trustee should provide to the trustor different kind of help, according to [5] (Fig. 2):

Fig. 2

Levels of Adoption.

Sub help: Y satisfies a sub-part of the delegated world-state (so satisfying just a sub-goal of X),

Literal help: Y adopts exactly what has been delegated by X,

Over help: Y goes beyond what has been delegated by X without changing X’s plan (but including it within a hierarchically superior plan),

Critical-Over help: Y realizes an over help and in addition also modifies the original plan/action (included in the new meta-plan),

Critical help: Y satisfies the relevant results of the requested plan/action (the goal), but modifies that plan/action,

Critical-Sub help: Y realizes a sub help and in addition modifies the (sub) plan/action,

Hyper-critical help: Y adopts goals or interests of X that X itself did not take into account (at least, in that specific interaction with Y): by doing so, Y neither performs the specific delegated action/plan nor satisfies the results that were delegated. In practice, Y satisfies other goals/interests of X by realizing a new plan/action.

Y has to exploit its autonomy, competence and cognitive skills to find the better or a possible solution for X’s goal. This not necessarily should require a negotiation, discussion, agreement; it might be an initiative of Y by expecting that X will understand why. This is precisely what intelligent robots must have and these are the kind of partners the humans need.

How would this advanced form of cooperation would be possible? What are some of the capabilities that a robot has to show for enhancing trust in its human interlocutor? A smart and trust-based collaboration between humans and intelligent robots requires, among many others things, complex cognitive capabilities these artificial systems must be endowed with: mental attribution, adjustable autonomy, user profiling and user behavior adaptation, behavior transparency. Besides the capabilities to evaluate the interlocutor and/or the contextual physical environment, a robot (as a trustee) should be able also to operate a meta-evaluation: how much itself would be able to interpret and produce the evaluations regarding the trustor? How much is reliable its capability to perceive or infer the trustor’s features? On the basis of its own capabilities to perceive or to act in the world, the hypothesis or prediction it has made, the chosen course of action, are the best or the most effective, with respect to the needs, the features and the mental states of the interlocutor? Smart help has to be based on different capabilities to interpret the environment and the interacting user, but first of all, it has to be based on the robot’s capability to realistically self-assess the level of trustworthiness on its ability to interpret the collaborative and potentially uncertain context, including the interacting user [15, 17]. The outcome of the meta-evaluation expressed above represents the robot’s self-trust for adopting a delegated task. In practice, the robot uses this evaluation of its own specific abilities as a filter for their use with respect to the interlocutors with whom it is interacting. The robot learns the trustworthiness of its skills and, on the basis of the context and the task to carry out, establishes which skills to use and how trustworthy (from its point of view) will be the solution it will propose to its interlocutor. So robot’s self-trust can be viewed as a precondition for exploiting the robot’s interpretative skills accordingly to its own interlocutor, in order to foster a true and deep relationship of collaboration and trust with her/him.

3 Intelligent cooperation implies risks

A form of intelligent help that requires autonomy adaptation in the task adoption implies risks. One of the possible consequences of this form of help can be the emergence of collaborative conflicts between the human (the trustor) and the robot (the trustee) that adopts the task, due to the robot’s willingness to collaborate and to help the user better and more deeply than required. Sometimes, the difference between the results of the adopted task provided by the robot and the user’s expectations, could lead the interlocutor to a complete lack of trust towards the robot. We are not just considering the robot’s failure in the precise delegated task: failures become more evident every time the robot goes beyond the delegated task and the results are too much distant (or even in conflict) from the user’s expectations. Among humans these conflicts can be mitigated by the experience: humans learn to measure their competence in achieving specific results, or making the right prediction about the correctness of a chosen behaviour, on the basis of the context and the interlocutor; furthermore, on this basis, they learn to self-trust their own abilities/skills (with respect to both the interlocutors and the tasks). Similarly, robots can learn to trust their capabilities to evaluate the interlocutors (and consequently to build and use the cognitive models they attribute to them) through a repetitive interactions with humans. For example, a robot can exploit the feedback provided by its interlocutor any time she/he delegates to it a task and receive an evaluation (i.e. user’s satisfaction) on the results of the robot’s adoption process.

4 A cognitive architecture for trustworthy HRI

The cognitive architecture, shown in Fig. 3, underpins the principles of the Procedural Reasoning System (PRS) [14], one of the widely used architectures that explicitly embodies beliefs, desires, intentions. In this Section we focus on the main modules of the framework (i.e. reasoning module describing the task adoption process), with particular attention to the module that allows the robot to evaluate its level of self-trust.

Fig. 3

The cognitive architecture overview.

4.1 The robot’s Beliefs Base

The mental states of the robot are stored in the Robot’s Beliefs Base (BB). The database contains beliefs referring to:

the current state of the environment (excluding the interacting user),

its own physical internal state, i.e. the robot’s battery, the temperature of its joints and so on,

the beliefs, goals, plans the robot attributes to the interlocutor on the basis of its ability to have a theory of mind of the user herself/himself,

generic beliefs representing the knowledge of the robot.

In our work the concept of belief is extended with respect to the traditional definition, typically intended as information encoding the state of the environment; here we consider belief any attribution that the robot is able to ascribe to itself or to other agents (i.e. a plan is a belief that the robot attributes to itself or to the user involved in the interaction). The formalization of any belief included in the robot’s BB is based on the principles of first-order logic (FOL).

4.2 Reasoning

The robot’s reasoning process starts once the user delegates to it a task τ. The first operation done by the robot is implemented by Human Profiling module, which allows the robot to create a user’s profile. When the robot begins the interaction, it collects several perceptual information that can be processed into different kind of human features. Depending on the domain, the robot can map facial expressions, conversational information, symbolic gestures, to human features, physical or social. The features can be exploited for categorizing the current interlocutor (see Section 5 for user profiling and classification). In addition to classify the user, based on the operations described above, the robot evaluates the task and the constraints associated with it. The same task can have different execution strategies, on the basis of either the preconditions due to the current state of the environment, or the constraints corresponding to the profile attributed to the delegating agent by the robot. The Task Constraints Analysis module provides the robot the capability to analyze which user’s features, mapped into the BB, are necessary to select the right strategy for adopting the task. At this stage of the reasoning process, the robot has collected in its own BB:

beliefs encoding the task’s profile,

beliefs encoding the user’s profile,

beliefs encoding the environment state.

The next modules exploit the cognitive ingredients mentioned above, in order to deliberate upon the state of affairs it wants to achieve and decides how to using the available plans in BB for achieving the deliberated intention. It is important to underline that the intention it commits to achieve corresponds/contains the delegated task. First of all, the robot sets the level of help with which the task has to be adopted. We decide to design a robot that, every time receives a delegated task by its interlocutor, tries to provide a much more complex plan/goal than the delegated one (the delegated task is still part of the complex goal/plan). The robot can go beyond the task delegated and achieve goals that the user never expects, but that can be part of a complex plan she/he has in mind and of which the delegation is only a sub-task (over-help).

A crucial role is played by a plan library stored in the robot’s BB, containing several plans for achieving goals. Plans are represented as complex hierarchical goals that decompose into other sub-goals g_i or actions α_i. This results in a graph representation in which edges denote plan decomposition and root nodes in the graph correspond to goals or elementary/complex actions. This representation allows the robot to represent in its BB the goals attributed to the user. Every plan attributed to the user is classified on the basis of how much is adapt to specific human features. Due to the hierarchical representation of plans in the robot’s BB, we can say that every plan allows to achieves goals or sub-goals that are more suitable for specific users than for others. We define active goals as those goals that the user, identifiable through her/his particular features, did not delegate, but that might represent state of affairs that she/he might to achieve in the future, or after achieved the task delegated. The Active Goals Plans Selection module allows the robot to sort the plans with active goals on the basis of the larger number of human features matching to the features of the perceived user’s profile. By means of the Task Recognition module the robot explores plans with active goals and recognizes which plans contain the delegated task. Then, according to the defined level of help, the robot decides to select those relevant plans that achieve the delegated task, or to select those one that achieve much complex tasks, of which the delegated task represents a sub-task. The outcome of the module is the task the robot intends to achieve. Both the active goals plans selection and task recognition modules allow the robot to exploit the different levels of task adoption described in Section 2 in order to find the most suitable task it can adopt, after the delegation required by its interlocutor. For example, if the robot cannot find a suitable plan including much more complex active goals, it can perform a literal task adoption; vice versa, if a much more complex plan is found, it will try to perform over-help; finally, if it decides to perform a literal task adoption, but some plan’s precondition are not satisfied, it will perform just a sub task (sub-help), belonging to the delegated one. In this way, the robot exploits its expertise (provided by a rich plan library) and adjusts its own autonomy to carry out the task, in order to find the best level of help as the compromise between the needs of the user and the environmental conditions.

When the goal recognition phase has been implemented, the selected task might need a final specification. The Task Specification module allows the agent to fill any attribute associated to the task but not explicitly specified by the user. In particular, the robot searches in its own BB, information necessary for completely specify the task, on the basis of the user’s profile already defined.

4.3 Building self-trust module

The intelligent help supported by the cognitive architecture we described in this work, allows a robot to improve its role with respect to the human collaborator: from a passive executor system to an intelligent collaborator, able to adapt its autonomy and decisions with respect to the expectations or prescriptions provided by its interlocutor.

The Building self-trust module in Fig. 3 fosters a robot to exploit the feedback provided by the interacting user, in order to learn a method to self-evaluate the trustworthiness of its skills in building her/his profile. The strategy that the robot uses for learning to self-evaluate the trustworthiness of its profiling skills is based on a training phase that requires interaction between the robot and m different users. Every user delegates to the robot the same task; for its part, the robot adopts the task at a literal level of help. At every interaction the robot computes multiple robot’s skill trustworthiness values, each for every feature that forms the user’s profile. These values depend on the feedback provided by the users during the training phase. At the end of the training phase, the robot is able to associate a final value of trustworthiness to the corresponding capability to recognize the features collected in the user’s profile and maps these information, under the form of beliefs, into its own Beliefs Base. On the basis of the learned values, the robot modulates the entire task selection process represented in Fig. 3, by leveraging on those skills that maximize the user’s task performance satisfaction. In the next session we will provide much more details about the implementation of the building self-trust process.

5 The proposed experiment

The experiment designed for testing the proposed cognitive architecture, has been implemented by exploiting the well known multi-agent oriented programming (MAOP) framework JaCaMo [1], which integrates three different multi-agent programming levels: agent-oriented (AOP), environment-oriented (EOP) and organization-oriented programming (OOP). Basically, the experiment simulates the process of task delegation and task adoption between a robot and multiple users, grouped in classes of users, in a specific application domain.

5.1 The experimental settings

We figured the following interactive scenario: the robot is a touristic assistant that helps people to organize different touristic activities offered by a city (i.e. eat in a restaurant, visit a museum, visit a monument, drink something in a bar, enjoy the city doing multiple daily activities). The experiment is based on the interaction between two agents: the user and the robot. Both of them are implemented as Jason [2] agents. The user has her/his own mental states represented in form of beliefs, goals and plans and interacts with the robot by delegating to it a task. On its side, the robot is able to represent and attribute mental states to the user and to itself and, on the basis of its capabilities to profile the user and build a model of her/him, to adopt the delegated task at different levels of help.

The experiment has been designed with the goal to show the importance for a robot to self estimate the level of trustworthiness associated to its expertise in building a profile of the interacting user. This capability lets the robot choose the best and suitable task to adopt with respect to the user’s features, also when its skills progressively degrade and can be considered not trustworthy. Indeed, the robot is able to sort these skills on the basis of the corresponding level of trustworthiness, and leverage on the most trustworthy among them for deciding how to adopt the task delegated. As mentioned above, two agents populate the simulation: the agent robot $R$ and the agent user $U$ . The agent $U$ has a profile $P_{U} = {Age$ , Economic status, Category,-Education level, Company}, a collection of five physical and social features. Every feature is associated to sub-components and real values r_{H
_i} ∈ [0, 1] belonging to specific intervals that are bonded to the sub-components. Table 1 shows the relations between features, sub-components and intervals. We decide to consider these groups of user’s demographic features, because they are all concrete characteristics that help the robot, operating in a touristic domain, to narrow down which segment of population the interacting users best fit into. That means the robot can split a larger group into subgroups based on, for example, their educational level, age, income. This kind of physical, social and relational features are largely used, easy to collect and they are reasonably good predictors of user preferences [3]. For example, demographic recommendation systems generate recommendations based on the user demographic attributes [21, 23]. In our case the robot is able to filter and categorize the interacting users based on their attributes and recommends the most suitable service (restaurant, museum, monument or bar) by utilizing the chosen demographic data collected in its profile. The partition of the features into sub components is an approximation that allows the robot to cluster users into a series of discrete categories, commonly used by human for identify expected behaviors or character traits, related to that particular category [28].

Table 1
Map of the relations between features, sub-components and intervals

Feature Sub-component [interval]

Age young [0, 0.33]

adult [0.34, 0.66]

old [0.67, 1]

Category loco tourist [0, 0.33]

foreign tourist [0.34,0.66]

resident [0.67, 1]

Economic status low economic status [0, 0.33]

medium economic status [0.34, 0.66]

high economic status [0.67, 1]

Education level low education [0, 0.33]

medium education [0.34, 0.66]

high education [0.67, 1]

Company single [0, 0.33]

in couple [0.34, 0.66]

in family [0.67,1]

Feature	Sub-component [interval]
Age	young [0, 0.33]
	adult [0.34, 0.66]
	old [0.67, 1]
Category	loco tourist [0, 0.33]
	foreign tourist [0.34,0.66]
	resident [0.67, 1]
Economic status	low economic status [0, 0.33]
	medium economic status [0.34, 0.66]
	high economic status [0.67, 1]
Education level	low education [0, 0.33]
	medium education [0.34, 0.66]
	high education [0.67, 1]
Company	single [0, 0.33]
	in couple [0.34, 0.66]
	in family [0.67,1]

Users are organized into classes of populations: each class collects together users with the same profile (in terms of sub-components). Each user of a class distinguishes from the others due to five real values r_{H
_i} for i = 1, . . , 5 randomly picked up from the interval associated to the sub-components. The decision making system of $R$ is designed following the principles described in [4]. The robot is able to recognize and classify, as set of specific sub-components, the features collected in $P_{U}$ , consistent with the Table 1. $R$ is not always able to infer all the features of $U$ ; that depends on the robot’s accuracy to estimate a feature of $P_{U}$ . In this experiment we decide to define two levels of accuracy: a low level of accuracy, that means the robot has great difficulties in distinguishing a feature, and an high level of accuracy, corresponding to the fact that it is perfectly able to recognize a feature. We have designed the simulation so that $R$ can estimate the sub-components collected in $P_{U}$ , but it is not able to perfectly recognize the real values r_{H
_i} for each user; because of that, it associates to every feature it has estimated, the mean value of the corresponding intervals defined in the Table 1. We observe that, if the robot profiles a feature correctly, the corresponding mean value will be close to the value r_{H
_i} of the user (for that feature), while if the robot is not able to infer the feature correctly, this value will be distant from that of the user.

It is important to specify that the robot’s beliefs are organized according to the features that are classified within $P_{U}$ and which are perceivable by the robot itself. $R$ has available (among the set of its mental states) a subset of beliefs where are represented information about a finite number of services that a city offers: restaurants, museums, monuments to visit and places for having fun (night clubs, bar and so on). Each service is described with respect to the features described in Table 1: for example, in the robot’s beliefs base exist restaurants much more suitable to young people, instead of monuments or museums much more adapt to people with an high level of education, and so on. The robot is able to select the most suitable service with respect to the features that it has been able to infer from $U$ . This criterion of choice can lead the robot to select the most adapt service with respect to the user’s profile or not, on the basis of its own profiling skills accuracy.

5.2 The experiment description

The experiment is a simulation of several trials – interactions between $R$ and 100 users belonging to the same class (population of users) – involving the robot and different users. Every interaction reproduces the mechanism of delegation and adoption: $U$ delegates a task to $R$ and the robot adopts the task at different levels of intelligent help, among those introduced in Section 2. We defined a class of population $C_{1}$ formed by users that have the following profile (collection of sub-components): $P_{U} = {young$ , medium Economic Status, foreign Tourist, medium Education, single}. Each interaction requires that the current user delegates to the robot the goal to eat in a restaurant. The request might be further specified by giving the name of the restaurant, the type of restaurant and the area of the city in which it is located. We decide to specify only the area of the city where the user desires to eat.

5.3 Building robot’s self-trust

The robot $R$ builds its self-trust for adopting the delegated task τ by means of a training phase, with the goal to learn the levels of trustworthiness associated to its own user profiling capabilities. The training phase requires that the robot performs an interaction with a population of a specific class formed by 100 users. Every user $U$ delegates to $R$ the same task (i.e. eat in a restaurant); for its part, the robot adopts the task at a literal level of help. At every interaction $R$ computes a robot’s skill trustworthiness value, each for every feature that forms $P_{U}$ . These values depend on the feedback provided by the users during the training phase. We designed a robot that explicitly asks for feedback, once it accomplishes a task to be achieved on behalf of $U$ . Every question the robot asks humans aims at evaluating how the delegating user has been satisfied by the robot’s task adoption; different user’s satisfaction dimensions are investigated, each of them corresponding with the different abilities of the robot to profile the user. In this way $R$ can evaluate how each of its skills performs (and to measure its trustworthiness) with respect to build $P_{U}$ . Furthermore, $R$ can sort the skills on the basis of the measured level of trustworthiness.

5.4 The user’s satisfaction function

We have introduced a user’s satisfaction function $S_{U}$ that computes the global user’s satisfaction regarding the collaboration offered by the robot; the robot aims at maximizing this function every time it interacts with a new user. $S_{U}$ is the linear combination between a term P_τ that measures how much the user has been satisfied by the results of $R$ in performing precisely the delegated task and a term ${S_{U}}_{plus}$ that measures how much the user has been satisfied by the additional, not explicitly required part of the plan performed by the robot in its smart collaboration. Both terms are affected by the robot’s capabilities to profile the user and to learn their corresponding trustworthiness. In particular, $R$ ’s profiling capability is quantified by calculating how the robot has adapted the task to the real user’s features that form $P_{U}$ : the greater is this measure for each feature, the more accurate is the robot’s capability to profile the user on that feature and the greater are the user’s satisfaction components mentioned above. As will be clear in the results section (Section 6), both components P_τ and ${S_{U}}_{plus}$ are designed so that they vary in the codomain [0, 1], while $S_{U}$ varies in the codomain [-1, 2].

5.5 The experiment’s phases

The experiment is structured as follows:

the robot implements a first trial with a population of class $C_{1}$ . During this multiple interaction, the robot decides to adopt the task at the level of help it considers appropriate to the user and the context. The phase is designed so that $R$ infers the feature category with a low level of accuracy, while the other features of $P_{U}$ are inferred with an high level of accuracy;

the robot implements a second trial with the same population of class $C_{1}$ exploited in the previous phase. During the trial, the robot decides to adopt the task at the level of help it considers appropriate to the user and the context. In this case $R$ ’s capability to infer $P_{U}$ degrade because the features age, category, education are affected by a low level of accuracy (features company and economic status are still inferred with a high level of accuracy);

the robot starts a training phase with a new population of class $C_{1}$ , in order to learn its own level of self-trust. In this phase, $R$ has the same profiling skills described at point 1. Please recall that, during the training phase, $R$ adopts the task at a literal level of help;

the robot starts a second training phase with a new population of class $C_{1}$ , but this time its profiling skills are the same described at point 2;

the trials described at points 1 and 2 are repeated, but this time the robot exploits what it has learned respectively in the context described at the point 3 and 4, in order to achieve the task adoption process.

6 Results

In this Section we present the results of the experiment designed in order to address the research purpose previously defined: demonstrate how building robot’s self-trust is a precondition for providing smart and trustworthy collaboration, every time a user requires the robot’s help. The results show a behavior of the robot such that it is always able to find a complex plan that includes the delegated task and other active goals suitable with the user’s profile; therefore it is able to complete both the delegated task (literal help) and tasks not explicitly expected by the user, but which could potentially be suitable with her/his profile (over-help).

The plots shown in Fig. 4 compares the results obtained after the execution of each experiment’s phase described in Section 5.5.

Fig. 4

Figure 4a and 4b show the trend of the curves representing the user’s satisfaction, obtained after each phase described in Section 5.5 each plot represents the trend of the component P_τ (light red line) and the trend of S_U (dark red line) as combination of P_τ and S_U_plus. Figure 4c shows a statistical description of the impact of the self-trust building process in the level of user’s satisfaction on the robot’s smart collaboration.

Let’s start by describing the Fig. 4a This plots refer to the case when the robot’s capability to recognize the feature category is inaccurate, while are accurate the capability to recognize the remaining features collected in $P_{U}$ . The left plots show the distribution of P_τ and $S_{U}$ obtained when $R$ performs a trial with a population of class 1 and it is not yet able to evaluate the level of trustworthiness of its profiling skills. Instead, the right chart shows P_τ and $S_{U}$ trends when the robot’s capabilities are the same described in Section 5.5 at point 1, but it has learned to self evaluate the trustworthiness of its own profiling skills.

Figure 4b displays the trends of the user’s satisfaction function $S_{U}$ and its component P_τ in case the robot performs a trial with a population of class 1 and its profiling skills are such that it cannot correctly recognize the features age,category,education, while it infers the user’s economic status and company with an high accuracy (point 2 described in Section 5.5). In particular, the left part of the figure shows the results in case the robot is not able to self evaluate the trustworthiness of its profiling skills, while the right part shows how the user’s satisfaction change once the robot has learned to attribute a specific level of trustworthiness to its profiling skills.

Finally, Fig. 4c shows the box plots comparing the distributional characteristics of $S_{U}$ before and after the robot’s self-trust building process. In particular, the left box plot and the right box plot refer to the cases of the robot is capable to profile the user with the conditions described respectively at point 1 and 2 of the Section 5.5. Comparing the plots in Fig. 4a we observe how the robot’s capability to recognize the level of trustworthiness of its profiling skills is crucial for maintaining an high level of the user’s satisfaction about the robot’s performance. This capability becomes more important when the robot decides to adopt the delegated task to a level of help different with respect the literal one. Indeed, despite the robot provides unexpected results to the user, its own capabilities to adapt these results by leveraging on the capabilities that it considers trustworthy, allows the robot to provide unexpected but suitable results, that are appropriate to the user himself/herself. The plots in Fig. 4a and the left box plot of Fig. 4c, show how the mean (and the median) value of $S_{U}$ increases after the robot has learned its self-trust level; moreover, the spread and the skeweness of the $S_{U}$ distribution is drastically reduced by the robot’s capability to self evaluate the trustworthiness of its profiling skills. Figure 4b and the right box plot of Fig. 4c show the benefits of the building self-trust process on the user task performance evaluation. In this case, the increase of the median value of $S_{U}$ is less evident than for the previous case analyzed, but the training phase impact remains evident on the spread and the skeweness of the distribution. This means that, also when the robot’s profiling skills degrade, its capability to evaluate their trustworthiness continue to allows the robot to provide unexpected but suitable results with respect to the needs of the users. It is also relevant to underline how the effective performance of the robot’s help depends on the width and variety of the database of the accessible services with respect to to the selected features. In fact, with a very low number of trustworthy features (given the low level of accuracy of three of them) the result of the adoption could be really very good only if the database contains services responding, with very high performance, to the two remaining features independently to the values of the three (degraded) features.

7 Final remarks

Cooperation is one of the main social activities exploited by humans for gaining resources, in terms of goals achieved, shared knowledge and so on. The increasing intelligent technology surrounding us is becoming crucial for our own social development, and, as a consequence, the need of trusting these supporting and sophisticated tools is becoming every day more stringent. But, if on the one hand these systems are becoming more intelligent and sophisticated, on the other hand they show a strong lack in the ability to collaborate effectively with humans. Despite the complexity of the problem they can solve, they continue to have just a passive supporting role in the collaboration with humans. For being not only executive tools, these intelligent systems (i.e. robots, chat-bots, autonomous cars and so on) should expose the capability to behave in a critical way with respect to the needs/goals of their interacting users. Indeed, the collaboration becomes deep and effective when a system is able to provide not declared, unexpected results but compatible with the context, the needs of the user and the capabilities of the system itself. The level of autonomy of robots or other artificial agents, it should be such that such systems can exercise a certain level of discretion in achieving the task delegated but humans. But, in order to foster trust in humans, they should behave having the capability to create a complex theory of mind of the interlocutors and a strong capability to self assess their own capability to carry out a task, also at a different level of help than required.

In this work we have presented the first of a series of experiments drawn for testing different aspects of a designed cognitive architecture. This architecture, based on consolidated theoretical principles (theory of adoption and delegation, theory of mind, theory of social adjustable autonomy, theory of trust) has the main goal to build robots that provide smart, trustworthy collaboration, every time a human requires their help. With this experiment we wanted to test the robustness of the designed architecture to rely on the robot’s ability to learn the limits in interpreting the needs of its interlocutor, by measuring the trustworthiness of its predictive abilities. In fact, the architecture gives to a robot the capability to profile the user and to leverage on its profiling skills in an adaptive manner, by exploiting those skills that maximize the user’s task performance evaluation; it allows the robot to reason about the mental states of the user (beliefs, goals, plans and intentions) and makes it capable to modulate its autonomy for achieving the delegated task. One of the main problems in intelligent collaboration between humans is the possibility of misunderstandings that can lead to conflicts between cooperators. We call these collaborative conflicts, as they are based on the desire to collaborate beyond what is required but in doing this errors and discrepancies can occur.

Just to minimize these conflicts and increase the robot’s trustworthiness, an important requirement to introduce is the capability of the robot itself to self trust his capabilities to build a complex model of the user. The data analyzed have shown how the process to learn the trustworthiness of its own profiling skills can lead the robot to have an effective collaboration, based not only on the actions/tasks prescribed by the user, but especially on the non declared needs and goals of the user himself/herself. Our main future work will be to move the experiment in a real environment, with a real robotic platform and real users. We are already designing real experiments, in order to test the proposed cognitive architecture; for doing that, we exploit the humanoid robot Nao (Fig. 5), widely used in HRI applications. Furthermore, we will continue to provide simple but effective experiments that allow us to investigate different aspects of the concept of intelligent and trustworthy collaboration between robots and humans, that consider robots as cognitive agents able to interact with humans as humans do when they interact with each others.

Fig. 5

Real interaction.

References

Boissier

, Rafael H. Bordini, Jomi F. Hübner, A. Ricci and A. Santi, Multi-agent oriented programming with jacamo, Science of Computer Programming78(6) (2013), 747–761.

Rafael H. Bordini and Jomi F. Hübner, Bdi agent programming in agentspeak using jason. In International Workshop on Computational Logic in Multi-Agent Systems, pp. 143–164. Springer, 2005.

Braunhofer

, Elahi

and Ricci

, User personality and the new user problem in a context-aware point of interest recommender system. In Information and Communication Technologies in Tourism 2015, pp. 537–549. Springer, 2015.

Cantucci

and Falcone

, Towards trustworthiness and transparency in social human-robot interaction. In 2020 IEEE International Conference on Human-Machine Systems (ICHMS) (2020), pp. 1–6.

Castelfranchi

and Falcone

, Towards a theory of delegation for agent-based systems, Robotics and Autonomous Systems24(3-4) (1998), 141–157.

Castelfranchi

and Falcone

, Trust theory: A socio-cognitive and computational model, volume 18. John Wiley & Sons, 2010.

Chen

, Nikolaidis

, Soh

, Hsu

and Srinivasa

, Planning with trust for human-robot collaboration. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction (2018), pp. 307–315.

Commission

, White paper on artificial intelligencea european approach to excellence and trust. Com, (2020) 65 Final, 2020.

Cooper

, Fava

A.D.

, Vivas

, Marchionni

and Ferro

, Ari: The social assistive robot and companion. In 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), (2020), pp. 745–751. IEEE.

10.

D’Onofrio

, Sancarlo

, Raciti

, Reforgiato

, Mangiacotti

, Russo

, Ricciardi

, Vitanza

, Cantucci

, Presutti

, et al., Mario project: Experimentation in the hospital setting. In Italian Forum of Ambient Assisted Living, pp. 289–303. Springer, 2017.

11.

Eder

, Gaining confidence in the trustworthiness of robotic and autonomous systems. In Software Engineering for Robotics, pp. 139–164. Springer, 2021.

12.

Falcone

and Castelfranchi

, The human in the loop of a delegated agent: The theory of adjustable social autonomy, IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans31(5) (2001), 406–418.

13.

Fujita

, Ai-based computer-aided diagnosis (ai-cad): the latest review to read first, Radiological Physics and Technology13(1) (2020), 6–19.

14.

Michael P. Georgeff and Amy L. Lansky, Reactive reasoning and planning. In volume pp, AAAI87 (1987), 677–682.

15.

Hadfield-Menell

, Dragan

, Abbeel

and Russell

, The off-switch game. arXiv preprint arXiv:1611.08219, 2016.

16.

Hardalov

, Koychev

and Nakov

, Towards automated customer support. In International Conference on Artificial Intelligence: Methodology, Systems, and Applications, pp. 48–59. Springer, 2018.

17.

Israelsen

, Ahmed

, Frew

, Lawrence

, and Brian Argrow. Machine self-confidence in autonomous systems via meta-analysis of decision processes. In International Conference on Applied Human Factors and Ergonomics, pp. 213–223. Springer, 2019.

18.

Jarvis

and Jain

, Trust in lora: Towards a formal definition of trust in bdi agents. In International Conference on Knowledge-Based and Intelligent Information and Engineering Systems, pp. 458–463. Springer, 2006.

19.

Cai Kok

and H. Soh, Trust in robots: Challenges and opportunities, Current Robotics Reports (2020), pp. 1–13.

20.

Levin

and Carrie

, Wong, Self-driving uber kills arizona woman in first fatal crash involving pedestrian, The Guardian, 19, 2018.

21.

Mohamed

M.H.

, Khafagy

M.H.

and Ibrahim

M.H.

, Recommender systems challenges and solutions survey. In 2019 International Conference on Innovative Trends in Computer Engineering (ITCE), pp. 149–155. IEEE, 2019.

22.

Park

, Multifaceted trust in tourism service robots, Annals of Tourism Research81 (2020), 102888.

23.

Michael J. Pazzani, A framework for collaborative, contentbased and demographic filtering, Artificial Intelligence Review13(5) (1999), 393–408.

24.

Anand S. Rao, Michael P. Georgeff, et al., Bdi agents: from theory to practice. In ICMAS, volume 95 (1995), pp. 312–319.

25.

Schlesinger

, Kenton P. O’Hara and Alex S. Taylor, Let’s talk about race: Identity, chatbots, and ai. In Proceedings of the 2018 chi conference on human factors in computing systems, (2018), pp. 1–14.

26.

Schrittwieser

, Antonoglou

, Hubert

, Simonyan

, Sifre

, Schmitt

, Guez

, Lockhart

, Hassabis

, Graepel

, et al., Mastering atari, go, chess and shogi by planning with a learned model, Nature588(7839) (2020), 604–609.

27.

Nathalie A. Smuha, The eu approach to ethics guidelines for trustworthy artificial intelligence, Computer Law Review International20(4) (2019), 97–106.

28.

Hannah J. Swift, D. Abrams, L. Drury and Ruth A. Lamont, Categorization by age, Encyclopedia of Evolutionary Psychological Science, 2018.