Abstract
To provide assistance and support to the elderly disabled and cognitively impaired, the recognition of their activities of daily living (ADL) must be accurate and precise with regards to the object use for the activity situations. Current knowledge-driven and ontology-based activity recognition techniques model object concepts from assumptions and common everyday knowledge of object use of routine activities. Modelling activities from assumptions and common everyday knowledge of object use could lead to faulty recognition of particular routine activities and possibly undermine abnormal activity trends. A significant step in the recognition of activities of daily living is the discovery of the object use for specific routine activities due to its ability to relate object use to their associated activities. The discovering particular object(s) which are used to perform routine activities could help enhance knowledge-driven ontology-based activity recognition with the object use for specific activities and the associated activities as ontology concepts. This paper focuses on the recognition of simple activities of daily living from object use and interactions in the home environment. We take advantage of the object use for routine activities discovered from a topic model process to augment activity ontology concepts for activity recognition. The experimental results obtained using the Kasteren and Ordonez datasets show it is significantly encouraging, comparable and improved on results published using the same datasets.
Introduction
Activity recognition is an important emerging area of research in pervasive computing due to its significance in the provision of support and assistance to the elderly disabled and cognitively impaired. It is a process involved in identifying what an individual is doing, e.g. Sleeping, Showering, and Making Food etc. Research efforts so far has focused on the use of video [1, 2, 3], wearable sensors [4, 5] and wireless sensor networks [6, 7] to monitor simple human activities. Video based activity recognition captures body images which are segmented and then classified using context based analysis. Unlike sensor based activity recognition, video based suffers from accurate segmentation of captured images which affects classification process. Sensor based activity recognition monitor object usage from the interactions of the objects in the home environment. Typically, sensor based activity recognition follows data-driven, knowledge-driven or a combination of both techniques to identify activities. Data driven activity recognition use machine learning and statistical methods which involve discovering the data patterns to make activity inferences. Research efforts by [8, 6, 7] show the strengths of the approach in the learning process. Inferences in most cases for data driven approaches are hidden and or latent, thus requiring activities identified to be expressed in understandable format for the end user. Again, data driven approach suffers from its inability to integrate context aware features to enhance activity recognition in the home environment. On the other hand, knowledge driven and ontology-based methods model activities as concepts, associating them to everyday knowledge of object use in the home environment through a knowledge engineering process. The modelling process involves associating low-level sensor data to the relevant activity to build a knowledge base of activities in relation to sensors and object. Activities are then recognised by following logical inference and or inclusion of subsumption reasoning. In comparison to data driven techniques, knowledge driven techniques are more expressive and inferences are usually in the format easily understood by the end user [9]. Knowledge driven techniques in most cases depend on everyday knowledge of activities and object use to build and construct activity ontologies. Knowledge of object use is mostly by assumptions, regular every day knowledge of what object are used for routine activities or even wiki-know-how [10].1 In this paper, we follow a sensor based activity recognition with sensors capturing object use as the result of the interactions of object in the home environment. We regard object use and interactions as atomic events leading to activities. With different objects, we could have different activities. In some cases, we could have activities with shared or similar object use. The approach we propose specifically follows object use as events and entries to recognise activities. This is in line with what is obtainable in real world situations and most home environments. Interactions of objects as object use in the home environment result to activities. These object use and interactions as atomic events result to particular routine activities. Our motivation of following this approach is such that knowledge driven activity recognition through an activity Ontology modelled and constructed from the everyday knowledge of object use may not fit into certain activity situations or capture specific routine activities in home environments but could be satisfactorily achieved with accurate and precise object use for specific routine activities. If an activity model has been developed based on the generic and or assumed knowledge of object use, the recognition model may fail due to activities and objects fittings which differs with individuals and home environments. Generic ontology models have been designed and developed as in Chen et al. [8] to emphasise re-usability and share ability. As a way forward, we argue that, it is essential to extend this ontology to accommodate object concepts that are specific to the routine activities with regards the individual and the home environment. To provide assistance and support to the elderly and cognitively impaired, the recognition of their activities of daily living (ADL) must be accurate and precise with regards to the object use events. This framework significantly comes in handy where object use for routine activities have not been predefined. So the challenge becomes discovering specific object use for particular routine activities. This paper focuses on the process of the recognition of ADL as a step towards the provision support and assistance to the elderly disabled and cognitively impaired. It also emphasises on knowledge driven techniques and shows that the knowledge acquisition process can be extended beyond generic and everyday knowledge of object use to build activity ontology. Given these, our work harnesses the complementary strengths of data and the knowledge driven techniques to provide solutions to the limitations and challenges highlighted above. In this context, the major aim of this paper is to extend knowledge driven activity recognition to include a process of acquiring knowledge of object use to describe contexts of activity situations. In particular, we seek to acquire knowledge by identifying activities within object data stream by activity-object use discovery. In addition to this, we represent this acquired knowledge ontologically to build an activity recognition prototype capable of accurately recognising activities of daily living. We propose a framework composed of a context description module and an activity ontology module as components. The context description module augments the activity ontology to acquire the knowledge of object use in the home environment through an activity-object use discovery enabled by the Latent Dirichlet Allocation topic model. The contributions of this paper are as follows.
Activity-object use discovery in context for the likely objects use for specific routine activities. We use topic model through activity-object use discovery to acquire knowledge for concepts formation as part of an ontology knowledge acquisition and learning system. Extend the traditional activity ontology to include the knowledge concepts acquired from the activity-object use discovery and context description which is significantly important especially in home environments and scenarios where object use for routine activities have not been predefined. The evaluation and validation of the proposed framework.
The remainder of the paper is organized as follows. Section 2 provides an overview of the related works, while Section 3 describes the proposed activity recognition approach. Section 4 provides experimental results based on the Kasteren and Ordonez datasets which were used to validate the proposed framework. Section 5 concludes this paper.
An overview of the activity recognition framework.
Activity recognition approaches can be classified in two broad categories data and knowledge driven approaches. This classifications are based on the methodologies adopted, how activities are modelled and represented in the recognition process. Data driven approaches can be generative or discriminative. According to [9], the generative approach builds a complete description of the input (data) space, usually using a probabilistic model. The resulting model induces a classification boundary which can be applied to classify observations during inference. The classification boundary is implicit and a lot of activity data is required to induce it. Generative classification models includes Dynamic Bayes Networks (DBN) [11, 12], Hidden Markov Model (HMM) [11, 13, 14], Naive Bayes (NB) [15, 16, 12], Topic model Latent Dirichlet Allocation (LDA) [17, 18]. Discriminative models, as opposed to generative models, do not allow generating samples from the joint distribution of the models [9]. Discriminative classification models includes nearest neighbour [19, 20], decision trees [21], support vector machines (SVMs) [22, 23, 19], conditional random fields (CRF) [24], multiple eigenspaces [25], and k-means [19]. With regards to the state of the art approaches discussed above, data driven approaches have the advantage of handling incomplete data and managing noisy data. Not all the data driven approaches have the advantage of handling temporal information. Another drawback associated with data driven approaches is that they lack the expressiveness to represent activities as they as learnt in the model. They need to be annotated and sometime needing elements of semantics for expressiveness. Topic models inspired by the text and natural language processing community have been applied to discover and recognise human activity routines in research works by Katayoun and Gatica-Perez [17] and Huynh [4]. Huynh [4] applied the bag of words model of the Latent Dirichlet Allocation (LDA) to discover activities like dinner, commuting, office work etc. The process involved activity discovery of partitioned sensor segments of time windows. Katayoun and Gatica-Perez [17] discovered activity routines from mobile phone data, Huynh [4] used wearable sensors attached to the body parts of the user. Activities discovered in both work were latent, lacked expressiveness and minimal opportunities to integrate context rich features. Our work also significantly differs from Katayoun and Gatica-Perez [17] and Huynh [4] with the modelling of the activity-object use using an ontology activity model. Knowledge driven ontology models follow web ontology language (OWL) theories for the specification of conceptual structures and their relationships. Ontology based activity recognition are an emerging area in knowledge-driven approach. It involves the use of ontological activity modelling and representation to support activity recognition and assistance. Ontology uses the formal and explicit specification of a shared conceptualization of a problem domain [26]. Vocabulary for modelling a domain is provided by specifying the objects and concepts, properties, and relationships. This then uses domain and prior knowledge to predefine activity models to define activity ontologies [9]. Latfi and Lefebvre [27] proposed an ontology framework for a telehealth smart home aimed at providing support for elderly persons suffering from loss of cognitive autonomy. Similarly, Marjan et al. [28] proposed and ontology based framework to recognise activities and the events occurring in the home. Chen et al. [9] and Chen and Nugent [29] proposed an ontology-based approach to activity recognition in which they constructed context and activity ontologies for explicit domain modelling. Sensor activations over a period of time are mapped to individual contextual information and then fused to build a context at any specific time point. Subsumption reasoning were used to classify the activity ontologies, thus inferring the ongoing activity. Knowledge driven ontology models also follow web ontology language (OWL) theories for the specification of conceptual structures and their relationships [30]. OWL has been widely used for modelling human activities for recognition, which most times involve the description of activities by their specifications using their object and data properties [29]. In ontology modelling, domain knowledge is required to encode activity scenarios, but it also allows the use of assumptions and common sense domain knowledge to build the activity scenarios that describe the conditions that drive the derivation of the activities [31]. Recognising the activity then requires the modelled data to be fed to the ontology reasoner for classification. The authors of [9, 31] followed generic activity knowledge to develop an ontology model for the smart home users. Whilst these approaches to model activities to depending on common sense domain knowledge and its associated heuristics are commendable, they may lead to faulty activity recognition due to lack of specificity of the object use and contexts describing the activity situations. Specific considerations to object use for routine activities could be been made just as home settings and individual object usage differs which may not be applicable to ontologies developed from generic knowledge of object usage. They also do not follow evidenced patterns of object usage and activity evolution as they rely on generic know hows and hows to to build ontology models. In view of these limitations, we apply an LDA enabled activity discovery technique to discover likely object use for specific routine to augment the generic ontology modelling process by [9] in our work. The framework we propose in this paper, also extends previous works by Ihianle et al. [32, 33, 34] by the inclusion of the ontology activity model.
Overview of our activity recognition approach
To achieve activity recognition, the proposed framework supports object use as contexts of activity situations through activity-object use discovery, information fusion of activity and object concepts, activity ontology design, development and modelling, and then activity recognition. The proposed framework is implemented as illustrated with the architecture in Fig. 1 – having a complementary topic model context description module to augment the traditional ontology driven activity recognition. The architecture is made of two component modules – the context description module and the ontology module. Object use for specific routine activities as activity-object use distributions and activity context descriptors are discovered using a Latent Dirichlet Allocation (LDA) topic model in the context description module. An activity ontology is developed using the discovered object use for the respective activities as ontology concepts in the ontology module. Activity recognition is achieved by observed object use query on the activity ontology for the relevant activity situations. As a unified framework, the functions of these component modules are integrated to provide a seamless activity recognition platform which takes in inputs of sensor and object use observations captured in the home environment representing atomic events of object interactions. We describe in detail the component modules in the following subsections.
Context description module
The context description module augments the traditional knowledge driven activity recognition framework [8, 29, 35]. Its main function is to provide the knowledge of object use for specific routine activities as activity-object use distributions and activity context descriptors. These object use for specific routine activities are the contexts describing the specific routine activity situations, hence the name context description module. To provide the basis for an activity recognition, the knowledge of object use for respective activity concepts are required. The necessity of these object use knowledge is such that activities as high-level events are a result of low-level tasks or atomic events of object interactions. Traditional knowledge driven activity recognition frameworks [10, 8] model ontologies from generic object use assumptions or every day knowledge of object use. However, the traditional (generic) models may not be the case in every home setting or environment as this may lead to erroneous object descriptions of low-level tasks and eventually incorrect activity recognition. To design and model activity situations ontologically, there must be an accurate process of acquiring the knowledge of object use or context descriptors which describe activities or maps to activity as higher level of events. The context description module performs its function by a process of activity-object use discovery and activity context description which uses the output from the activity-object use discovery process. We briefly describe this modular process below.
Activity-object use discovery by Latent Dirichlet Allocation
The activity-object use discovery is dependent on the Latent Dirichlet Allocation (LDA) introduced by Blei et al. [36]. The LDA generatively classifies a corpus of documents as a multinomial distribution of latent topics. It takes advantage of the assumption that there are hidden themes or latent topics which have associations with the words contained in a corpus of documents. It then requires the bag of words (documents) from a corpus of documents as input and number of topics as a key parameter. In the context of our activity-object use discovery process, the activity topic number and bag of sensor observations corresponds to topic number and bag of words respectively of the LDA. We explain determining the activity topic number and bag of sensor observations below.
Activity topic number: A key parameter needed by the LDA process is the topic number. In the context of the activity recognition we propose in this paper, the number of activities corresponds to the topic number. The activities to be recognised adds up to the activity topic number. For experimental purposes we use the number of activities specified in the dataset.
Bag of object observations: The bag of objects observation we propose is analogous to the bag of words used in the LDA text and document analysis. In text and document analysis, a document (bag) in a corpus of texts can be represented as a set of words with their associated frequencies independent of their order of occurrence [37]. Disregarding the order of word occurrence, the bag of words is a representation of the words in the document with their frequencies. We follow the bag of word approach to represent discrete observations of objects or sensors of specific time windows generated as events in the use or interaction of home objects. In this regard, we refer to it as bag of object observations. To satisfactorily achieve bagging of the objects accordingly, the stream of observed sensor or objects data are partitioned into segments of suitable time intervals. By this, the objects and the partitioned segments then respectively corresponds to the words and documents of the bag of words. If a dataset is given by
The observed objects
An overview of the activity ontology module.
The LDA takes advantage of the assumption that there are hidden themes or latent topics which have associations with the words contained in a corpus of documents. It also involves the use of bag of words in the corpus of documents which are generatively classified to latent themes or topics and word distributions. We conversely apply this assumption to the activity-object pattern discovery context that latent activity topics would have associations with the features of object data in the partitioned segments of the bag of sensor observations discussed above. The documents are presented in the form of objects segments
Given the assumptions above, the activity-object patterns can be calculated from the Eq. (4) below:
where
In the context of the activity recognition, modelling activity concepts for recognition would rely on the probabilistic distribution of the objects given the activity topics. The LDA topic model,
The ontology module is composed of the knowledge base as a repository of information consisting of the modelled activity ontology concepts, data, rules used to support activity recognition. Just like other knowledge bases, it functions as a repository where information can be collected, organized, shared and searched. The activities and the context descriptors from the context description module are designed, developed following description logic, knowledge representation and formalism and then added to the knowledge base. The knowledge base is made of the TBox, ABox and the reasoner (see Fig. 2 for an overview of the activity ontology module). The TBox is the terminological box made of the activities concepts and the relevant context descriptors of object use as defined and encoded as ontology concepts. The ontological design and development process gradually populates the TBox by encoding the activities and context descriptors from the context description module as ontology concepts. The ABox is the assertional box made of the instances and individuals of the concepts encoded in the TBox. They are asserted through properties which may be object or data properties. For all the terminological concepts in our TBox, instances and individuals of these are asserted through different properties to populate our ABox. In addition to the activities and context descriptor concepts and instances, we also added temporal concepts and instantiated them following the 4D fluent approach to allow for a realistic reflection activity evolution and transition. Also based on their temporal properties as usual time of occurrence, activities could be modelled as static and dynamic activities. The resultant activity ontology created with the fusion of likely object use and behavioural information from the activity context descriptions makes it possible for activity inferencing. The reasoner checks the relationships between the concepts in the TBox and also checks the consistencies in the ABox for the individuals and instances to perform activity recognition by information retrieval. The eventual result from the information retrieval are the activities or activities situations.
Ontology concepts used
Ontology concepts used
Ontology notations used
The process of modelling an activity concept resulting from a set of sensors and object use requires asserting all the objects concepts and with their times to be encoded to represent the activity situation (see Tables 1 and2 for a list of concepts and notations used in our analogies). If an activity situation Breakfast is the result ofMicrowave_On and Fridge_On at times t1 and t2 respectively, then Breakfast can be asserted with the properties hasUse and hasStartTime as hasUse(Microwave_On, hasStartTimet1), hasUse(Fridge_On, hasStartTimet2). The activity is therefore modelled as a list of the objects and with their times ordered temporally. The example of Breakfast from Microwave_On and Fridge_On at t1 and t2 can then be encoded by the expression Eq. (6) below.
Typically, activity situations or activities in the home environment are a result of specific objects use. To model activity situations accurately, it is important to extend the traditional activity ontology modelling to include specific resources and or objects use for the specific routine activity. The activity context descriptors resulting from the activity-object use discovery forms the resources and objects use concepts to be modelled onto the activity ontology for the specific routine activities such that:
Activities: The activity topics are annotated as activity concepts analogous to the activity situations in the home environment. This represents a class collection all types of activities set as
Objects: These represents class collection of all objects as activity context descriptors in the home environment set as
4D-fluents with activities and resources. Recall 
If the function
The expressions below encodes enhanced sensors or object outputs with their temporal attributes.
The activity context descriptors are then modelled as resources and objects class concepts accordingly and then added to the ABox so that:
In the home environment, activities are performed differently, in different ways and times within the 24 hour day path. Some of these activities can be performed – at specific times of the day making them have static times occurrence (Static activities), performed at different or varying times of the day (Dynamic activities), and in some cases, same or similar objects may be used to perform some of these activities. We make our analogy using Breakfast, Lunch, Dinner, Toileting and Showering as examples of activities in the home environment. Breakfast, Lunch and Dinner are examples of different activity concepts which can be performed with same or similar object interactions given that they are food related activities. In difference, they have specific times of the day they are performed making them static activities. Given their similarities, they can be modelled as subclasses of the activity Make Food, however, they differ with regards to their respective temporal properties. Whilst they inherit all the properties of Make Food by subsumption, they can be easily confused in the recognition process if modelled in the ontology without consideration to their usual times of performance. Distinction can only be achieved for them by the specification of the time intervals they are usually performed. On the other hand, activities like Toileting and Showering can be performed at any time of the day making the process of distinguishing them less dependent on their temporal properties, hence, they are dynamic activities. Dynamic activities are not constrained within any time interval. We therefore extend the ontology of activity situations to include static and dynamic activities using the 4D-fluent approach [38], requiring the temporal class concepts Timeslice and TimeInterval to be specified using the relational properties tsTimesliceOf and tsTimeIntervalOf respectively as illustrated in Fig. 3, the time intervals Interval1 and Interval2 holds the temporal information of the time slices for the static and dynamic activities respectively. An instance of a TimeSlice of an activity whether static or dynamic is linked by the property tsTimeSliceOf and property tsTimeInterval which then links this instance of TimeSlice with an instance of the class TimeInterval.
Modelling a static activity: A static activity is modelled by requiring the specification of the TimeSlice, TimeInterval class concepts and with the context descriptors for that activity. The activity concepts described above are extended so that the hasUse object property encodes the usage of the objects for the static activity by specifying the static activity as the domain class concept and ranges to all the object classes which describes the context descriptors. We further extend this with the temporal properties which requires tsTimeInterval to have domain TimeSlice and Resources and it ranges TimeInterval to capture specific time interval of the day through Interval (a sub class of TimeInterval). The time instants of the activity are captured through the tsTimeSliceOf with domain TimeSlice and Resources and it ranges to TimeSlice. With regards to Fig. 3, the expression Eq. (11) encodes a static activity so that Interval1 asserts the time interval of the day the static activity is performed using the object
The expression Eq. (12) then asserts Interval1 to cover the time instant
Modelling a dynamic activity: Similar to static activities, it is modelled by requiring the specification of the TimeSlice, TimeInterval class concepts and with the context descriptors for that activity. An instance of a TimeSlice of a dynamic activity is linked by the property tsTimeSliceOf and property tsTimeInterval and then links this instance of the class TimeSlice with an instance of class TimeInterval which may be Interval2. Interval2 ranges to cover the full 24 hour cycle of the day as asserted by expression Eq. (13).
The expression Eq. (14) then asserts Interval2 to cover the time instant
A sample of sensor status and output
The activity recognition process is enabled by the Algorithm 3.3 which performs a mapping of the activity situation using the observed objects. A comparison is made through reasoning by the ontology to retrieve the closest activity situation described by the contexts of object observed as sensor data. The activity recognition uses object use query like constructs adapted from the Temporal Ontology Querying Language (TOQL) [38] on the knowledge base to retrieve activity situations fitting the requirements of the query. As an advantage, sensor states and status of object use as implemented in the activity ontology can be used in queries to reflect real situations of object usage in the home environment. A typical query is comprised of SQL like construct (SELECT-FROM-WHERE) for OWL which treats the ontology classes and properties like database tables and columns. An additional AT construct in the query compares the time interval for which a property is true with a time interval or instant. Considering the scenario in the home environment where sensor status and outputs captured are reported as given in Table 3. The Algorithm 3.3 enables the activity recognition process. The inputs are observed sensor along their time lines as
an activity
Considering the scenario of observed objects in Table 3, the question would be “What activity does these sensors or object use in Table 1 represent at the particular time?” The query construct to provide the activity recognition is as given in the schema Eq. (15).
To validate the framework presented in this paper, we used the Kasteren et al. [7] and Ordonez et al. [6] datasets captured in two different home settings with similar events and activities (see Tables 4 and 5 for an overview of the home setting descriptions and activity instances). Our choice of these dataset was driven by the fact that the Kasteren and Ordonez dataset contain a lot of sensor activations as object use with dense sensing applied. Different types of sensors (for example pressure sensors, magnetic sensor etc tagged to home objects like microwave, dishes, cups) were used to capture object interactions representing the different activities. To further enhance the learning process, the activities contained therein have been performed in varied ways and accurately annotated in the ground truth.
We followed a 4-fold cross validation on the datasets.
Our criterion for evaluation is to compare recognised activities with the ground truth provided with the dataset based on the average true positives TP, false positives FP and false negatives FN per activity. The results are then further evaluated based on precision, recall and F-score.
Home setting and description
Home setting and description
Activity instances in the Kasteren and Ordonez dataset
Activity concepts and the discovered context descriptors for Kasterens house A
Visualisation of some activity-object distributions for Kasteren house A.
To generate the context descriptors for routine activities, we followed the context description process described in Subsection 3.1 above. Recall the LDA process requires activity topic numbers and the bag of object observations. We used the number of activities as given in the dataset. Next, we partitioned the dataset using 60 seconds sliding windows to construct the bag of object observations in the form of a segment-object-frequency matrix as we have described previously Subsection 3.1 above. For the LDA activity-object use discovery process, we used the constructed bags of object observations as inputs, the activity topic numbers from the datasets and we set the dirichlet hyperparameters
Activity Concepts and the discovered context descriptors for Ordonez house A
Activity Concepts and the discovered context descriptors for Ordonez house A
Visualisation of some activity-object distributions for Ordonez house A.
Common object concepts for Kasteren and Ordonez houses.
Unlike the 7 activity set in the Kasteren A ground truth, we discovered 6 activity topics in this process indicative of the activity sets which we annotated as Leaving, Toileting, Showering, Sleeping, Make Food and Drink. Our Make Food activity in this case represents Breakfast and Dinner due to same and similar context descriptors or object usage. We distinguish these activities through the ontology static activity modelling. This is similar for Ordonez A with 7 activities annotated as Leaving, Toileting, Showering, Sleeping, Make Food, Spare Time and Grooming. Make Food for the Ordonez house A represents Breakfast, Lunch and Snack.
To this point, we have generated the context descriptors for the various activities for Kasteren A and Ordonez A houses. To facilitate activity recognition and eventual evaluation of the framework, we model the activities and the context descriptors from the previous Subsection in an ontology activity model. To enhance and support a unified ontology model and with common concepts shared and which can be reused across the similar home environments, we developed unified activity ontology for the Kasteren and Ordonez datasets as illustrated in Fig. 6. The green coloured rounded rectangles represents common object concepts in both homes. The blue rounded rectangle has been used specifically Ordonez concepts which are not shared in Kasteren concepts. This unified ontology model can also be extended and adapted further for similar homes thus reducing the amount of time taken to construct and develop activity ontologies.
Common activity concepts for the Kasteren and Ordonez houses.
The activity concepts were also modelled accordingly, but due consideration was given to Make Food which represented a group of activities. The Make Food activity as it implies, corresponds to a group of activities involving making of food and ranges from Breakfast and Dinner. With regards to the Kasterens dataset, we class Breakfast, and Dinner as static activities with super class Make Food sharing same or similar context descriptors and also they are performed at specific times of the day. To further enhance shared ontology concepts and reuse, we harmonised the Kasteren and Ordonez activity concepts as illustrated in Fig. 7 onto the activity ontology to form a set of unified activity concepts. Similar to the object concepts, we have colour coded activity concepts in this unified set of activity concepts with static and dynamic activities as super classes so that the green rounded rectangle represents common activity concepts, blue rounded rectangle as activity concepts in the Ordonez house and not in the Kasteren House and the red rounded rectangle as activity concept in the Kasteren house and not in the Ordonez House. As part of our proposed framework, we added instances and individuals to the of the object concepts making the model more expressive (assertions used in populating the ABox) for example instantiating Microwave with Microwave_On to suggest the state of the object or sensor when in use. The ABox was further populated with assertions using object and data properties as we explained in Subsection 3.2 incorporating the context descriptors for the activity situations preparatory for activity recognition. The modelled activity situations or concepts are then linked to their respective context descriptors as object states through the properties as assertions added to the ABox. Activity recognition is enabled by an object use query to retrieve activity situations based on observed sensor or object data similar to the schema Eq. (15). The Algorithm 3.3 implements the recognition process.
Activity recognition performance for Kasteren A and Ordonez A
Activity ontology and implementation of an object based query.
To facilitate execution of activity inference, the modelled activity ontology is imported into the java based TOQL environment as illustrated in Fig. 8. Object based queries are executed by mapping the observed objects and its temporal information from the dataset to the closest activity in the imported activity ontology through ontological reasoning.
We evaluated the proposed framework, thus allowing us to compare the activities inferred and recognized by our framework with the ground truths. Leaving, Toileting, Sleeping and Showering for both datasets were recognised with significantly high results as shown in Table 8. This performance can be attributed to the discovered object use and context descriptors for these activities. For these activities object use were accurately specific with minimal false positives. The process of discovering likely the object use for routine activities significantly ensured that these activities were associated to the objects used to perform them. Breakfast and Dinner for Kasteren house and Breakfast and Lunch for Ordonez house showed lower performance due to confusions from same and similar object use with Drink and Snack respectively. These activities have shared same and similar object interactions as observed with context description process hence been classed under the super activity Make Food. Recall that to further distinguish Breakfast and Dinner they were modelled as Static activities given the specific time of the day they are performed. To enhance their recognition, time interval properties and concepts enabled by 4D fluent approach were included. They were often recognised concurrently and led to high false positives in the process. However, the results achieved for them are quite encouraging. Overall, the average precision, recall and F-Score with the datasets as illustrated in Fig. 9 show impressive performance.
For further performance evaluation, we present the learning performance of our proposed activity recognition framework. Activities in the home environment can have different ways of being performed or the object use for activities may differ. A robust activity recognition model should have the ability of recognising activities irrespective of the object use and interactions. We evaluate the learning capability at the activity level further using the ground truth as the basis of evaluation. This comparison would involve the number instances of the different activities across folds at the activity level. A good model should be able to return almost the same number of activity traces as in the ground truth. The results as presented in Table 9 is indicative of good performance of the learning process of the framework. Leaving, Toileting, Sleeping and Showering were recognised with significantly high instances an minimal difference with the ground truth for both datasets. Breakfast, Lunch and Dinner showed lower performance due to confusions from same and similar object use with Drink. The number of correctly recognised instances in comparison to the ground truth is also very encouraging.
Summary of correctly recognised activity instances for Kasteren and Ordonez datasets
Summary of correctly recognised activity instances for Kasteren and Ordonez datasets
Average precision, recall and F-Score for Kasteren A and Ordonez A datasets.

We compared the results we obtained with the results reported by Kasteren et al. [7], Ordonez et al. [6], Ye [39], Riboni et al. [35] with ours for the Kasteren and House A as illustrated in the Fig. 10. Experimental methodology differs for the reported works. Kasteren et al. [7], Ordonez et al. [6] and Riboni et al. [35] all performed the evaluations using a leave one day out methodology. Ye [39] used a 10 fold validation just as we have used a 4 fold validation methodology. Although Ye [39] did not use timeslices which lead to improved results, we have used 60 seconds timeslices similar to Kasteren et al. [7], Ordonez et al. [6] and Riboni et al. [35]. Comparing our results directly with these other methodologies, our work performed significantly better with 91.9% for the F-Score. It is assumed the weaker performance reported by Kasteren et al. [7], Ordonez et al. [6] and Riboni et al. [35] might be due to the effect of evaluation methodology a leave one day out which meant fewer day representation for the object data. Comparing our results with Ye [39], we achieved a slightly higher F-Score which meant our proposed frame work for recognition is robust and significantly good.
Summary and conclusion
The activity recognition enhanced using topic model we propose in this paper provides the basis to learn and recognise activities. We carried out experiments using the Kasteren and Ordonez datasets. We evaluated the performance of the framework to recognise activities. In addition to the experiments, we compared our results to the results published using the same dataset in other literature. Based on the experiments and evaluations, we discuss the benefits and limitations of the proposed approaches.
Activity-object use and context description process: As part of the framework, we proposed the acquiring knowledge of object use by the object use discovery and activity context descriptions for the activities. However, activities like Breakfast, Lunch and Dinner sharing same or similar object use are considered as activity situations which can be made distinct by modelling them as static activities in the ontology. The main benefit of this process is its ability to discover unique object use as context descriptors for the activity situations. Limitations may arise for other similar activity situations like Drink and Snack as we observed with the datasets.
Performance of the activity recognition process: Experiments carried out on the datasets suggest good recognition performance for activities. Although the performance was encouraging for most activities, recognition were confused for activities sharing same and similar object use. Notably in this case was Drink and Snack which we modelled as dynamic activities and Breakfast, Lunch and Dinner modelled as static activities. Given the general performance of this activity recognition process as illustrated with figure 8, the activity recognition process on the average is significantly comparable.
Model learning performance: The aim of this evaluation was to assess the model learning ability. From the results, the contexts descriptors which led to activity situations in the ground truth are similar to the contexts descriptors we discovered hence the result achieved at the activity level. We obtained almost the same number of activity traces for the datasets in comparison with the ground truth suggesting good and significant learning.
With the experiments, assessments and evaluations using publicly available datasets, we can say that, i) The process of activity-object use and context description of activity situation provides accurately the needed object and activity concepts for the ontology modelling process. ii) Modelling activities as static and dynamic activities helps to improve activity recognition especially for activities with same and similar object interactions. iii) Given the results from the activity recognition process in comparison with other results published using the same datasets, we conclude it is significantly good and encouraging. The experimental and evaluation process using these datasets suggests that the features, components and the entire activity recognition process have been fully verified.
The future work should involve extending the ontology activity model by the consideration of more contextual features for the different types of sensing devices. It should also consider extending the temporal entities and features for progressive activities as they evolve.
Footnotes
