Handling temporality in human activity reasoning

1 Introduction

Human-aware Artificial Intelligent systems are goal directed autonomous systems that are capable of interacting, collaborating, and teaming with humans. The perceptions obtained by these systems can be useful for activity reasoning, intention recognition, activity verification, and activity support. This article tackles the problem of reasoning about activities a human is performing considering the temporality and durability of the activities and possible overlappings between them.

Nieves et al. [11] and Morveli-Espinoza et al. [10] used argumentation for determining inconsistent (and therefore different) activities from a set of hypothetical fragments of activities (henceforth, hypothetical fragments). In their approaches, all the perceived hypothetical fragments are analysed together as they happen at the same time and without considering their durability. However, the activities a human performs may happen at different times and have different duration, which should be reflected in the reasoning about human activity. In order to better understand the problem, let us present the following scenario. In this scenario, a man –let us call him Bob –arrives to his home after working and performs different activities until he goes to sleep (Figure 1 shows the different activities Bob performs from 17h until 6h). The activities are monitored and can be perceived in form of hypothetical fragments (we can associate an activity with the notion of goal and hypothetical fragments with sub-goals), which have to be compared in order to distinguish the activities.

OPEN IN VIEWER

Fig. 1

Bob’s activities after work from 17h until 6h.

Assume that there is a set of hypothetical fragments for the activity cooking and a set of hypothetical fragments for the activity taking a shower. Although both activities can be considered conflicting because a person cannot be frying something and at the same time taking a shower (conflicting actions), both of them were performed by Bob considering that they were performed in different time intervals. By applying the approaches proposed by Nieves et al. [11] and Morveli-Espinoza et al. [10], both sets of hypothetical fragments are analysed as they occur at the same time, which is not correct because they belong to different time intervals. The result is that only one activity could be performed, which is not correct. Thus, it is important to analyse both conflicts between activities and the temporal relation between them.

We can also notice that some activities partially (or completely) overlap, this means that there are hypothetical fragments that belong to the same time interval like the hypothetical fragments of talking with mom and cooking; however, there is no conflict between them, although they belong to different activities. In this case, another mechanism for distinguishing the activities has to be used. Let us assume that the hypothetical fragments of talking are detected at the same time of the hypothetical fragments of sleeping. This may lead to emerging conflicts since it is not normal to talk consciously when one is sleeping.

Against this background, the research questions that are addressed in this work are:

How to model the temporality constraints between human activities in the settings of activity reasoning? and

In addressing the first question, we will use Allen’s interval algebra [1] in order to represent the durability of the activities and the temporal relation between them. Regarding the second question, we will use hypothetical fragments for representing both an action and its context in terms of related observations (about both the human and the environment) and the goal that can be achieved by performing such action. Based on the elements of its context, emerging conflicts can be identified. We can consider hypothetical fragments as arguments. Hence, we can apply formal argumentation techniques for determining consistent sets of hypothetical fragments, which in turn will determine different activities or sets of non-conflicting activities (this is called local selection). Since temporality is taken into account we will base on Timed Abstract Framework (TAF) [4], which considers that arguments are valid during specific time intervals. Given that we also consider that attacks occur in some time intervals, we will extend TAF to support it. Besides, we propose a global selection, which aims to determine the degree of fulfillment or non-fulfillment of a given activity. Both types of selections aim to recognize the activities a human is performing.

The rest of the paper is organized as follows. Section 3 presents a short introduction about Allen’s algebra and TAF. Section 4 presents basic concepts on argumentation-based activity reasoning. Section 4 extends TAF for supporting activity reasoning and presents local selection. Section 5 presents global selection. The main properties of the proposed approach are studied in Section 6. In Section 7, we show the application of our proposal to the previously described scenario. Section 8 presents a discussion about the limitations, characteristics of the proposed approach, and compares it with some related work. Finally, Section 9 summarizes this article and outlines future research.

In this section, we present the main concepts about the interval algebra of Allen and about the Timed Abstract Framework.

3 Building blocks

In this section, we present the building blocks definitions. These are the human activity framework and the hypothetical fragment of activity. These definitions were extracted from [11].

We start by presenting the logical language that will be used throughout the article. Let L be a first order logic language used to represent the mental states of the agent, ⊢ stands for the inference of classical logic, ⊤ and ⊥ denote truth and falsum respectively, and ≡ denotes classical equivalence. We use lowercase roman characters to denote atoms and uppercase Greek characters to denote formulas, such that an atomic proposition b is a formula. If b is a formula, then so is ¬b. If b and c are formulae, then so are b ∧ c, b ∨ c, and b → c. Finally, if b is a formula, then so is (b).

Next definition models the human mind. We will follow the structure of the beliefs-desires-intentions (BDI) model [2].

Definition 6.(Human Activity Framework) An human activity framework ActF is a tuple of the form 〈 T , H A , G , O , Acts 〉 where:

T ⊆ L is a first order logic theory. T A denotes the signature of T ;

Given a human activity framework, one can build small pieces of knowledge which give hypothetical evidence of the achievement of a given goal by considering a set of beliefs (i.e., a set of formulas), ahypothetical action, and a set of observations of the world. These small pieces of knowledge will be called hypothetical fragments of activities:

Definition 7.(A Hypothetical Fragment of an Activity) Let ActF : = 〈T, H _A , G, O, Acts〉 be a human activity framework. A hypothetical fragment of an activity is represented by F : = 〈 K , O ′ , a , g 〉 such that:

K ⊆ T , O ′ ⊆ O , a ∈ H A and g ∈ G ;

4 Local selection

In this section, we introduce a framework to reasoning about human activity with respect to time. To this end, we study how to extend the TAF approach in order to support activity reasoning.

5 Global selection

Selecting hypothetical fragments by considering argumentation semantics is only one of the steps of activity recognition. An argumentation semantics can only suggest multiple competing sets of hypothetical fragments which could suggest the fulfillment of some activities. Therefore, we require a global selection of hypothetical fragments. By global selection, we mean a selection able to suggest:

degrees of both fulfillment and non-fulfillment of activities, and

Since activities are constituted by goals, we need to know what goals are associated to each extension. Let us recall that in an A-TAF, extensions are sets of hypothetical fragments t-profiles, so we use HYPOFRAGS (S) to denote the set of hypothetical fragments of an extension S . Given that a hypothetical fragment is always associated with a goal, a set of hypothetical fragments can be regarded as a set of goals. To this end, let us define the corresponding set of goals as follows: S G = { g | 〈 K , O , a , g 〉 ∈ HYPOFRAGS ( S ) } .

Considering that a set of hypothetical fragments can be regarded as a set of goals, the status of an activity is defined as follows:

Definition 16. (Status of Activities) Let ActF = 〈T, H _A , G, O, Acts〉 be a human activity framework, Φ = 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A-TAF with respect to ActF, and SEM be an argumentation semantics. An activity act ∈ Acts is:

achieved iff act ⊆ S G for all S ∈ SEM ( Φ ) .

It is important to observe that an extension S ∈ SEM ( Φ ) represents hypothetical fragments that argue why a particular activity is fulfilled. Considering the number of goals of each activity, we can define different degrees of achievement w.r.t each activity. Indeed, we can define a degree of achievement and a degree of non-achievement.

Definition 17.(Degree of (non-)achievement) Let ActF = 〈T, H _A , G, O, Acts〉 be a human activity framework, Φ = 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A-TAF with respect to ActF, SEM be an argumentation semantics, and act₁, act₂ ∈ Acts such that act₂ ⊆ act₁:

act₁ is (i/n)-achieved if act₂ is achieved w.r.t SEM (Φ), i = |act₂| and n = |act₁|.

6 Properties of the approach

This section studies some properties of our approach. These properties aim to describe how the behavior of the attack relation impacts on the semantics and how, in turn, it impacts on the results of the activity reasoning.

Proposition 2 states that the resultant conflict-free sets are the same in both approaches if there are not weak attacks. This means that all the pair of attacking hypothetical fragments have a strong attack relation in A-TAF, which is equivalent to the attack between arguments in a TAF. Let us recall that the attack relation att of a TAF is equivalent to the strong attack relation att _s of a A-TAF.

Proposition 2. Let 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A- TAF. If att = att _s , then cf_A-TAF = cf _TAF .

Proof. By reductio ad absurdum. Assume cf_A-TAF ≠ cf _TAF . This means that ∃ E ∈ cf A - TAF such that E ∉ cf TAF . According to TAF definitions, F and F′ have an attack relation in all the intervals they belong, that is, τ_(F,F′) = Av (F) ∩ Av (F′). On the other hand, if E ∈ cf A - TAF , it means that there is not an attack relation between F and F′ in the activity intervals in that extension. However, according to A-TAF definitions, it does not mean that they do not attack in other activity intervals or that they never attack. In the former case, this means that τ_(F,F′) ⊂ Av (F) ∩ Av (F′), that is, (F, F′) ∈ att _w and, therefore, att ≠ att _s , which contradicts the premise of the proposition and in the latter case (F, F′) ∉ att, which also contradicts the premise of the proposition.

Proposition 3 states that when there is a weak attack relation between at least one pair of hypothetical fragments in a A-TAF, then the set of conflict-free sets in TAF is a subset of the set of conflict-free sets in A-TAF. This means that when the temporal attack relation is weak, there are more amount of consistent hypothetical fragments, that is, the person has done more activities, including those that may be conflicting. Since the attack is weak, such activities were done sequentially.

Proposition 3. Let 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A- TAF. If ∃ (F, F′) ∈ att _w , then cf _TAF  ⊂ cf_A-TAF.

Proof. By reductio ad absurdum. Assume that cf _TAF  = cf_A-TAF. This means that att = att _s as proved above. Hence, all (F, F′) ∈ att _s and ∄ (F, F′) ∈ att _w , which contradicts the premise of the proposition.

Finally, Proposition 4 states that when there is a weak attack relation between two hypothetical fragments, then the t-profiles of both hypothetical fragments can belong to a conflict-free set considering those activity intervals where they do not attack, that is, where they are consistent. As in Proposition 3, this means that the performed activities were sequential. On the other hand, it may happen that two conflicting hypothetical fragments are not sequential, in such case, their t-profiles cannot make part of any conflict-free set.

Proposition 4. Let 〈HF _ActF , conf, TEMPREL, Av, att〉 be an A- TAF, F, F′ ∈ HF _ActF , and E ∈ cf A - TAF . If (F, F′) ∈ att _w and ( F , Av ( F ) ′ ) , ( F ′ , Av ( F ′ ) ′ ) ∈ E , then Av (F) ′ ∩ Av (F′) ′ ⊈ τ_(F′,F), where Av (F) ′ ⊂ Av (F) and Av (F′) ′ ⊂ Av (F′).

Proof. By reductio ad absurdum. Av (F) ′ ∩ Av (F′) ′ ⊆ τ_(F′,F). This means that there is an attack relation between F and F′ in at least one activity interval of Av (F) ′ ∩ Av (F′) ′. Hence, such activity interval (or intervals) cannot make part of the t-profile of both hypothetical fragments in the same extension. This contradicts one of the premises of the proposition.

7 Application to the scenario

In this section, we apply our proposal to the scenario presented in the introduction section. Indeed, this is a large scenario and we only take into account the necessary elements for illustrating the conflicts and attacks.

In this scenario, we focused on the temporal reasoning. For this reason we assumed that all the hypothetical fragments of each activity are known; however, it is not a rule. Indeed, formal argumentation is an appropriate technique for dealing with incomplete knowledge.

Let ActF _bob  = 〈T, H _A , G, O, Acts〉 be the Human Activity Framework for Bob’s activities, where:

T = { r 1 , r 2 , r 3 , r 4 , r 5 , r 6 , r 7 , r 8 , r 9 , r 10 , r 11 } ,

The intended meaning of the grounded atoms and is presented below:

The rules we are considering are:

r₁ = in _ the _ kitchen (X) ∧ grab (X, pan) ∧ frying (X, Y) → brown (Y)

From ActF _bob , the following set of hypothetical fragments can be constructed: HF _ActF  = {F₁, F₂, F₃, F₄, F₅, F₆, F₇, F₈, F₉, F₁₀, F₁₁}. Table 2 presents the details about them and Figure 2 shows these hypothetical fragments and their temporal relationship. For this example, we will work with three activity intervals: [17, 18], [18, 22], and [22, 6].

OPEN IN VIEWER

Fig. 2

Recognized hypothetical fragments.

Now, let us present the emerging conflicts between the hypothetical fragments:

Let us denote with A-TAF _bob the A-TAF for reasoning about the activities of bob, The next step is to apply a semantics, in this case, the preferred semantics will be applied. The resultant preferred extensions are:

S 1 = { ( F 1 , { } ) , ( F 2 , { [ 17 , 18 ] } ) , ( F 4 , { [ 17 , 18 ] } ) , ( F 5 , { [ 18 , 22 ] } ) , ( F 6 , { [ 18 , 22 ] } ) , ( F 7 , { [ 18 , 22 ] } ) , ( F 8 , { [ 18 , 22 ] } ) , ( F 11 , { [ 18 , 22 ] } ) , ( F 3 , { [ 22 , 6 ] } ) , ( F 9 , { [ 22 , 6 ] } ) , ( F 10 , { [ 22 , 6 ] } ) }

S 2 = { ( F 1 , { [ 17 , 18 ] } ) , ( F 2 , { [ 17 , 18 ] } ) , ( F 4 , { [ 17 , 18 ] } ) , ( F 5 , { [ 18 , 22 ] } ) , ( F 6 , { [ 18 , 22 ] } ) , ( F 7 , { [ 18 , 22 ] } ) , ( F 8 , { [ 18 , 22 ] } ) , ( F 11 , { [ 18 , 22 ] , [ 22 , 6 ] } ) , ( F 3 , { [ 22 , 6 ] } ) , ( F 9 , { [ 22 , 6 ] } )

In A-TAF _bob , we can notice that (F₁₀, F₁₁) ∈ conf and there is an attack relation between them in interval [22, 6]. For this reason, the t-profile of F₁₁ in S 1 only considers that it happens in interval [18, 22]. However, the same hypothetical fragment has a different t-profile in S 2 considering that it happens in intervals [18, 22] and [22, 6]. Only considering the conflict relation, these hypothetical fragments would never make part of the same extension. Nevertheless, from a temporal point of view they have an attack relation in the third activity interval; however, when F₁₁ occurs in the second interval and F₁₀ in the third one, there is not an attack relation between them.

The second step of our approach considers de global selection. The set of goals associated to each extension is the following:

S 1 g = { g 1 , g 2 , g 3 , g 4 , g 5 , g 7 , g 8 , g 9 , g 10 , g 11 , g 12 }

Let us now present the status of the activities:

Achieved activities:cooking, talking _ with _ mom, watching _ netflix, eating, taking _ shower, and talking

Finally, let us determine the degrees of fulfillment of the activities:

cooking is 2/2-achieved w.r.t. S 1 and S 2

In this section, we will discuss some aspects and limitations of our proposal. Besides, we compare it with some related work.

In this work, we have assumed that we can group hypothetical fragments in activity intervals such that these activity intervals do not overlap. This can be clearly seen in the example because a set of hypothetical fragments occur between 17 and 18 hours, the other set between 18 and 22 hours, an the last set between 22 and 6 hours. However, it may occur that there is always an overlapping between all the perceived hypothetical fragments. This is a limitation of our approach, and we plan to deal with it in future research.

The attack that emerged in the application scenario was between hypothetical fragments F₁₀ and F₁₁ that corresponded to activities sleeping and talking, respectively. After applying the semantics, we can notice that it is accepted that only one of the hypothetical fragment may happen. However, in this case it may happen that a person can talk while he is sleeping or he can stop sleeping to talk. Therefore, we need to further study this behavior and how to deal with it for improving the results.

Although the main contribution of our proposal is focused on activity reasoning, the fact of distinguishing conflicts from attacks can also be important for formal argumentation. In TAF or AAF approaches, the nature of attacks is not discussed and in structured argumentation (e.g., ASPIC+ [9], ABA [6], Delp [7]) an attack is generally related to the logical inconsistency. However, as we could analyse, in temporal activity reasoning conflicts are indeed related to logical inconsistency whereas attacks are determined using the conflicts and also the temporal relation between two hypothetical fragments. Thus, these works suggest that the nature of the attacks may depend on the contexts of the argumentation reasoning is carried out.

According to Ryoo and Aggarwal [12], statistical approaches that make use of models such as Bayesian Networks, Hidden Markov Models, and Conditional Random Fields, achieve good results with sequential activities; however, they require large learning datasets and fail to deal with temporal constraint management. Other approaches use reasoning in order to deal with temporal constraints. McKeever et al. [8] base on evidence theory to incorporate time related domain knowledge into the reasoning process. Stevenson and Dobson [13] tackle the problem of reasoning about concurrent activities. They make use of the Pyramid Match Kernel algorithm to support reasoning on recognising activities of varying grained temporal constraints. The difference with these approaches is that they do not deal with defeasible knowledge and in the former work, they do not tackle the problem of concurrent activities.

9 Conclusions and future work

This article presented an approach for activity reasoning, which uses Allen’s algebra to model the temporal constraints between hypothetical fragments of activities and extends TAF approach to support defeasible activity reasoning. The temporal and the conflict relation between hypothetical fragments determine the attack relation between them and the kind of attack relation. We demonstrated that our approach allows to distinguish between different activities by considering the activity intervals their hypothetical fragments belong.

Some future research directions were presented in previous section. Some other future research are: (i) we have used the Allen’s algebra for representing the temporal relation between hypothetical fragments. However, there are more elements of this algebra that can be used like the composition operation. It will be interesting to further study in order to determine how to apply them to activity reasoning,(ii) we plan to include uncertainty in the elements of the hypothetical fragments, and (iii) we also plan to use machine learning techniques for obtaining the necessary data for generating the hypothetical fragments; in this sense, we can complement both techniques for a better performance of the approach.