A foundational ontology-based model for human activity representation in smart homes

Abstract

Human activity detection within smart home (SH) is one of the basis of unobtrusive wellness monitoring of a rapidly aging population in developed countries. Most works in this area use the concept of “activity” as the building block with which to construct applications such as healthcare monitoring or ambient assisted living. This paper presents a model for human activity representation. The model has been developed by using the NeON methodology. The proposed human activity model consists of a network of ontologies classified in three categories: user ontologies, SH context ontologies and ADL ontologies, supporting user modelling, SH context modelling and ADL modeling, respectively. Furthermore, the formal definition of the main concepts and properties as well as a proof-of-concept ontology evaluation with a specification scenario are presented. The adoption of DOLCE+DnS Ultralite (DUL) ontology as an upper ontology aims to achieve a high degree of reusability and interoperability within heterogeneous smart home applications.

Keywords

Activity modelling ontology semantic interoperability smart homes e-health services

1. Introduction

Recent advances in sensing, networking and ambient intelligence technologies have resulted in a rapid emergence of smart environments. Among these, smart home (SH) has gained a lot of attention for the provision of enhanced quality of life within home. One of the motivations for smart home research is the significant increase of worldwide aging population. In Europe, the proportion of the EU27 elderly population above 65 years is predicted to rise to 30% in 2060 [9].

The elderly have specific health issues that have to be considered. A significant proportion of elderly population suffer or may suffer with higher probability from age-related conditions such as Parkinson’s disease, diabetes, cardiovascular disease, Alzheimer’s disease, different chronic diseases or limitations in physical functions. For them, SH technologies may provide assistance services by monitoring home environment and the elderly’s activities helping to enhance quality of life, prolong independent living and reduce caregivers’ necessary time and healthcare cost in general.

Research objectives in this area range from low-level data acquisition by sensors up to high-level context knowledge integration and inference through data-driven approaches and knowledge-driven approaches. Activity of Daily Life (ADL) in smart homes refers the activity relating to self-care and domestic tasks performed by the inhabitant routinely. The capacity of performing ADL without assistance from other people can be considered as a reference for the estimation of the independent living level of the elderly. The ADL includes self-care activities, such as sleeping, bathing, dressing, eating or taking medicine, and domestic tasks, such as preparing meals, housekeeping or watering plants. There are different types of ADL, but the common ground to all of them is that they should be recognizable as such by a non-technician (e.g. “prepare a meal”, “take a bath” or “watch television while sit on the sofa”). If human activities are correctly and automatically identified, a wide range of applications and services become possible, such as detection of health emergencies [14], recommendation services for correct performance of activity steps [2], professional advice on routine lifestyle [12], anomaly detection [20] or help in treatment prescription [19].

The human activity modelling is a challenging task in the development of smart home systems. On one hand, a variety of factors should be taken into consideration to describe an activity, such as location, physical object, time, etc. On the other hand, the different individuals perform one activity in different ways, or one individual may perform several activities in sequence or concurrently, which are known as “composite activities”. Thus an activity may be represented in diverse forms that depend on an individual’s preferences, leading to various ADL variants. This ADL diversity should be captured by the models, as a solution, context plays a core role in human activity modelling. The context is defined by Dey [7] as “any information that can be used to characterize the situation of an entity. An entity is a person, place or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves”. Semantic modelling has gained increasing attention due to its efficiency in context representation. The well-designed ontologies related to context modelling have been aligned with the DOLCE+DnS Ultralite (DUL) ontology [8], which has proved to be a foundational modelling basis for ontologies. They provide a large number of resources for reusing the concepts and statements to facilitate knowledge sharing and interoperability among smart home applications in an open and extensible manner.

In this paper, the human activity model based on foundational ontologies is proposed to present the corresponding contextual information. The main contributions of this paper are:

An exploration of various perspectives on contexts that used to describe the human activity in smart home in order to emphasize the key factors for multiple levels of context representation.

A human activity ontology is proposed as an approach to model the attachment of the sensor data and context knowledge to the abstraction level representation.

A proof of concept implementation and evaluation of the proposed model with a specific scenario in smart home domain.

The adoption of the concepts and statements from the widely reused ontologies promotes a high degree of interoperability and reusability within smart home applications.

The remainder of the paper is structured as follows: Section 2 presents the existing work in user and activity modelling. Section 3 illustrates the motivation for the proposed human activity ontology with a highlighted scenario. Section 4 introduces the NeON methodology that adopted in this paper as a guideline to develop the ontology; Section 5 presents the reused concepts and statements from existing ontologies and the core components for constructing the human activity ontology; Section 6 describes the implementation of the proposed ontology in Protégé and the application of the ontology to the specified scenario presented in Section 3. Finally, Section 7 concludes the paper and discusses the future work.

2. Related work

Currently, there are two main approaches for user and activity modelling in smart environments: data-driven approach and knowledge-driven approach. Data-driven approaches learn the activity models from datasets collected from diverse sensors via data mining and machine learning methods. This process mainly involves two phases: the construction of a probabilistic or statistical activity model and the establishment of mappings from sensor data inputs to outputs. Hidden Markov Model (HMM) [27], Conditional Random Fields (CRF) [26] and Dynamic Bayesian Networks (DBN) [17] are commonly used data-driven approaches to model activity and support temporal relationships representation. Data-driven approaches have as advantages the better handling of uncertain knowledge and the possibility of using rich period knowledge to capture long-time temporal dependencies. However, they suffer from the cold start problem, since large datasets are needed to train the model for each activity. They also lack in interoperability and reusability within heterogeneous smart home systems.

Knowledge-driven approaches model activities by using knowledge engineering methods and representation formalisms. Among the various knowledge-driven approaches, the ontology-based approach has gained increasing attention due to its powerful representation capabilities and reasoning methods for handling heterogeneous and imprecise sensor data. Generally, the expressive needs in context modelling related to activity detection include the ability to represent user-related information, hierarchical structure of environmental context and complex relationships among context instances (e.g. using temporal restrictions). Ontology-based modelling is a symbolic context representation method for describing knowledge about the context. The formalized vocabulary provided by an ontology can be used to define the concepts and properties, which facilitates knowledge sharing and reusing. Various ontology languages have been developed to describe these concepts and the relationships between them, such as Resources Description Framework (RDF) and Ontology Web Language (OWL). These Description Logic-based markup languages allow information to be read and processed by machines.

In current literature, various ontologies are proposed to model and represent context knowledge in smart environments. Vallerand et al. [28] propose an ontology model of contextual information aimed at the deployment of software services into the most suitable devices. To do so, they take into account user profiles (focusing on their interaction capabilities, built around the notion of interaction modalities, and preferences), device profiles, software profiles and environment topology. In [25], a novel ontology is integrated into an adaptation framework for situation-based and self-adaptive applications in smart environments. This ontology is embodied to model context, security, performance management and dynamic models for performing run-time reasoning and adaption. Even if the authors mention the possibility of representing activities, they do not center their description on this objective. Daoutis et al. [6] consider contextual information including different sensors, but they heavily center on visual perceptions obtained by cameras. Their purpose is to allow natural communication between robotic systems and humans. They obtain representation of the objects located in home together with their properties, as far as they are significant for the human-robot interactions. All these ontologies use specific concepts and statements tailored to concrete application areas. Because of this concreteness, they lack the ability of facilitating knowledge sharing and interoperability among various smart environments applications in an open and extensible manner.

There exist a number of ontologies widely accepted and reusable for modelling the related context knowledge across different applications. COBRA-ONT [30] is an ontology to describe the attributes and relationships related to people, space and activities. SOUPA [3] defines not only core concepts that describe information in common scenarios (e.g. person, time and space), but also extended concepts for narrower domains (e.g. home, office and entertainment). Event Model F [22] is an ontology that supports a comprehensive representation of the participant, mereological, causal and correlative relationships between events. These ontologies provide common knowledge concepts and statements, which facilitate knowledge sharing and interoperability. Our work takes advantage of this and reuses some of the common concepts and statements in these ontologies for the more specific domain knowledge representation in smart home area.

Regarding the specific domain knowledge of this paper, Chen et al. [4] propose seven ontologies to model semantic data in a smart home, related to physical equipment, activity, space, actor, medical information, software components and time. Okeyo et al. [18] extend existing ontology-based approaches by combining ontological and temporal knowledge representation formalisms to provide a powerful representation capability for composite activities modelling. Horst et al. [11] propose an ontology approach to bridge the gap between raw sensor data and high-level services in independent living applications by using a method of structuring knowledge and reasoning. The main contribution of their approach is the usage of generalized events, which makes modeling context more generic and simplifies context structuring and management. These works are more focused on activities modelling, as this paper is. However, they do not take user profiles into consideration during the inference process, while users are in fact the focus point in most smart home applications.

Several contributions have been made for modelling user profiles. Chen et al. [5] model smart home domain knowledge at two levels of abstraction, generic activity knowledge and user profile (the specific way to perform an activity). Skillen et al. [23] propose a user profile ontology to provide the personalization for context-aware assistive services, the rules are created by SWRL and used alongside the user profile for the purposes of application personalization. Martin et al. [21] combine ontologies that represent information based on a user profile along with machine learning techniques to infer user preferences automatically. The proposed profile uses two types of information: static information, such as name and age, and environmental user preferences that are configurable by the user. These user preferences allow the development of context-aware applications for the home environment that adjust proactively taking into account both user preferences and the particular situation.

The work in recent literature that is most aligned with this one is found in [16], in which two ontology patterns are described based on DOLCE+DnS Ultralite (DUL) for complex activity modelling in an extensible manner. Their ontology consists of activity specialization and activity composition patterns which allow the modelling, respectively, of both specialized activities and composite activities with interpretation queries. Our work is inspired by this one adding two new aspects: the first one is that we take the user profiles into consideration, which make smart home applications more adaptive, especially for the requirements of the elderly in the healthcare domain. The second one is the adoption of the NeOn methodology to provide the guidelines for every task in each development phase, particularly focusing on reusing the existing knowledge resources.

3. Motivation

The relevance of activity monitoring for healthcare services in smart home domain has been explored from the perspective of a collection sources of information. McAvoy et al. [15] highlight the key factors that include user, object, location, application, event, natural element and temporal for the context-aware applications within smart environments. Zhang et al. [31] explore the relevant concepts include person, event, activity, location and temporal for ontological ADL modelling in an assisted environment. In this section, the different perspectives for activity monitoring in smart homes that motivates this work are introduced.

3.1. User modelling

The user in smart home applications refers the owner of context and ADL information. The relevant information for user modelling mainly contains static contexts and dynamic contexts: A user’s static contexts represent the information that does not, or rarely change in time, such as the user’s name, age, gender and the role that a user plays. Conversely, the dynamic contexts refer the changing information, for example, the current location of a user, the heart rate record of a user and the specific way that a user performs an activity. These dynamic contexts could be modelled as user profile and preference to express the user’s state, and further used for providing personalized services.

3.2. SH context modelling

In smart home, the user performs activity in a diversity of contexts, which mainly include: spatial contexts, temporal contexts and measurement contexts [5]. Spatial contexts contain the locations information (e.g. bathroom, kitchen, bedroom) and physical object information (e.g. stove, telephone, television). Temporal contexts refer the time information or duration. Measurement contexts contain the state changes or digital records from the sensors, such as the state changes from the motion sensor in the dining room, the temperature in the room and the heart rate from the ECG sensor. The contexts are highly correlated with the description of user and ADL. For instance, the presence of the user in bedroom is detected by the state changes of motion sensor and the activity “preparing breakfast” occurs in the kitchen in the morning. These contexts are captured via multiple and heterogeneous sensors deployed in smart home, hence, the sensor modelling is the core element in the SH context modelling.

3.3. ADL modelling

ADL in smart home refers the activity relating to self-care and domestic tasks that performed by the inhabitant routinely, the capacity of performing ADL without assistance from other people can be considered as a reference for the estimation of the independent living level of the elderly. The ADL includes self-care activities, such as sleeping, bathing, dressing, eating and taking medicine, and domestic tasks, such as preparing meals, housekeeping and watering plants.

ADL is specified with related contextual information, e.g. occurring time, space, related object and posture. For instance, “preparing breakfast” normally is undertaken in the kitchen interacted with the stove in the morning, “sleeping” normally occurs in the bedroom related to the bed in the night. Moreover, an ADL can be performed by different individuals in different ways, for example, using different types of physical objects, with different sequence of actions and in different places within the house. The specific way of an individual performing an activity could be modelled as user activity profile.

According to diverse complexity and time interval, ADL could be modelled at multiple levels of granularity. For example, the coarse-gained activity “prepare drinks” can be split into fine-gained activities such as “prepare tea” and “prepare coffee”. Furthermore, there exist temporal relations between activities. Some composite activities can be performed by a sequence of simple activities, and activities may occur one after the other or concurrently or interleaved with each other. These complex temporal relationships between activities can be modelled based on qualitative temporal knowledge, e.g. sequential, concurrent and interleaved relationships, in order to represent composite activities.

3.4. Scenario

An example of the smart home applications is the activity monitoring use case for the elderly. Considering the following scenario:

Maria is an old woman with Parkinson disease who lives alone at a smart home, Carlos is her health expert and Juan is her son as the caregiver. Maria’s house is already equipped with different embedded sensors such as motion detection sensors, RFID, temperature sensors, etc. She is also attached with some wearable sensors such as accelerometer and ECG. When she performs the activities, the data generated by these sensors, which indicate Maria’s personal information and environmental information, and then send to the Carlos and Juan over a network. Carlos and Juan want to know what she does though a day, if she is in the normal situation when she performs these activities (e.g. Maria sleeps with the normal heart rate), if she performs activities in her preference (e.g. Maria prefers watch TV in the dining room with light off). And considering her Parkinson disease, they also want to know if she takes medicine before goes to sleep and takes medicine in a correct way (starts with opening medicine box and finishes with drinking water). These information will be used as a reference for Carlos and Juan to provide corresponding service when it is necessary.

4. Ontology development method

The main specific goal is to create an ontology that is capable to present information related to user, context, activity and relationships among them. For this purpose, the NeON methodology [24] is chosen due to its advantages of the flexibility and adaptability to diverse development scenarios. The NeON methodology is a scenario-based methodology that addresses the definition and formalization of ontology. It provides clear guidelines for every task in the development phase, and particularly focuses on reusing/reengineering the existing knowledge resources. Moreover, the NeON toolkit (http://neon-toolkit.org/) is also provided to support the technological availability.

According to NeOn methodology, a set of 9 scenarios are taken into consideration:

Scenario 1: From specification to implementation.

Scenario 2: Reusing and reengineering non-ontological resources.

Scenario 3: Reusing ontological resources.

Scenario 4: Reusing and reengineering ontological resources.

Scenario 5: Reusing and merging ontological resources.

Scenario 6: Reusing, merging and reengineering ontological resources.

Scenario 7: Reusing ontology design patterns.

Scenario 8: Restructuring ontological resources.

Scenario 9: Localizing ontological resources.

The scenario 1 is a basic and mandatory scenario that can be combined with the rest of scenarios. The development process is based on the NeON methodology, which defines the following tasks to be accomplished: (1) identification of purpose, scope and intended users of the ontology that have drove its development and implementation language, (2) analysis of ontology requirements, including non-functional requirements and functional requirements and (3) grouping the requirements with competency questions.

4.1. Ontology purpose and scope

A proper modelling of the entities and their associations in the form of ontology facilitates collaboration among diverse smart home applications. The smart home applications need to extract useful context information to express the performed activities and their relationships from the heterogeneous sensor observations and context knowledge. There are several goals that a human activity ontology should achieve. Firstly, the related contextual information should be represented as a whole, e.g. the information on sensor measurements and their sensing time and location, the user information, as well as physical objects attached to the sensors; Secondly, the different types of activities, the related context interpretation and their complex temporal relationships should be supported; Thirdly, the ontology should have interoperability and reusability to achieve a better degree of knowledge sharing within different smart home applications.

It is important to define what concepts should be included as well as what concepts should be excluded in this model. The ontology is proposed to model concepts used by the activity monitoring applications, it mainly focuses on the representation of user information, related context information and ADL information. The represented information could be viewed as a reference for the health experts to make decision on healthcare services, however, the concepts of activity recognition method and healthcare service, such as activity predicting, remaindering and human computer interaction, are not included in this model. Moreover, considering the ADL is concerned in smart home, thus the concepts of user, context and activity involve external environments, such as office and hospital, are exclude in this model.

4.2. Ontology implementation language

To formally represent the contextual information from raw data acquired from sensors to high-level ones such as activities, the Web Ontology Language (OWL, http://www.w3.org/2001/sw/wiki/OWL) – a Description Logic (DL) based markup language, is adopted to implement the human activity ontology. The OWL is defined by the World Wide Web Consortium (W3C) for the Semantic Web, it enables to formally specify the context data semantics to support the exchange of the context data among heterogeneous entities. The conceptual ontology is constructed based on classes, properties and individuals. The Protégé (http://protege.stanford.edu/), developed by Stamford University, is used as the implementation tool, this tool handles concepts strongly related to OWL-DL. The detailed description of the implementation of human activity ontology is provided in Section 5.

4.3. Intended end-users

The intended end-users include: the elderly who uses the sensors and performs activity to provide information and the health expert/caregiver who analyzes the elderly’s state by using these information.

4.4. Ontology requirements

The ontology requirements consist of non-functional requirements and functional requirements. In the following, the ontology requirements are analyzed in detail.

4.4.1. Non-functional requirements

The non-functional requirements refer the ontological principles, which work as the guidelines of how the human activity model needs to be designed in order to be applicable among various applications. According to NeON methodology, the ontological model needs to be carried out following four principles: (1) Interoperability. In order to establish a common understanding in a particular domain that can be applicable among various smart home systems, a mechanism is needed to ensure interoperability with respect to the syntax as well as semantics. (2) Modularity. The ontological model needs capture different structural knowledge, the portions use of the model could decrease the complexity of the system and simplify the development phase. (3) Reusability. The different systems concern different aspects of activities, thus the ontological model needs to support the reuse of its modules that imported by different domains. (4) Extensibility. The ontological model needs to be extensible for developing new functions and anticipating the use of shared vocabularies.

4.4.2. Functional requirements

The functional requirements aim at specific domain functions. Although several issues related to the functionality and concepts of the ontology are explained, the following overviewed analysis of functional requirements can facilitate the understanding of competency question included in this section:

The human activity model must support user modelling. The information about a user, not only the static information such as the name, the health data, the role that the user plays in the application, but also the dynamic information, such as the location of the user and the performed activity, should be described by the ontology. For example, in the scenario, Maria plays the role of the actor of an activity, her heart rate is recorded and she has the preference of watching TV in the dining room with light off.

The human activity model must support SH context modelling. To establish the sources of information, sensors are critical for providing various data and linking other entities. For example, in the scenario, the motion sensor links the place within smart home, the accelerometer links the user who wears it, and RFID sensor links the physical object that interacted with a user.

The human activity model must support activity modelling. The sensor observation collected from various sensors deployed in smart home should be mapped explicitly in order to model the activity information. In the scenario, the pressure sensor attached to the bed is triggered indicates the activity “sleeping” is performed by the user.

The human activity model must represent the activity pattern of the user. The concerned information of a user can be added as an interpretation of an activity to express the activity pattern of a user. In the scenario, the heart rate record when Maria is sleeping is concerned.

The human activity model must represent the temporal relations between activities. The representation of the temporal entities (e.g. activity valid time and duration) and processing of temporal relations between activities (e.g. an activity occurs before another activity) should be supported.

Based on these functional requirements, the competency questions are categorized into five groups:

CQG1. Competency questions about user modelling. (What is personal data of the user? What role does the user play? What is the user’s profile? What is the user’s preference?)

CQG2. Competency questions about SH context modelling. (What sensor is deployed in smart home? What is the sensor observation? Where is the sensor located? What physical object is the sensor attached to? Whom is the sensor attached to?)

CQG3. Competency questions about activity modelling. (Who performs the activity? Where does the activity occur? What physical object is related to the activity occurring? Does the happening of an activity have any constraints? What are the time characteristics of the activity?)

CQG4. Competency questions regarding ADL pattern modelling. (What information can be added as an interpretation to an activity pattern? What type is this activity in the given activity pattern?)

CQG5. Competency questions regarding the activities’ temporal relations modelling. (Is there temporal relation between activities? What kind of temporal relation between activities?)

5. Design phase

In order to fulfill the aforementioned requirements, a research is conducted on existing formalized foundational ontologies. Base on this, the human activity model is proposed to interlink a set of ontologies and describe the different subdomains needed for context knowledge modelling. The human activity model is developed by importing the exist foundational ontologies and extending statements on them according to the requirements.

5.1. Reusing existing ontologies

After the analysis of relevant ontologies related to the requirements in user and activity modelling domain, it is hard to find a single ontology that can fully cover the aforementioned requirements. However, there exist relevant ontologies and ontology design patterns, which can be partly reused for the construction of the human activity model. Moreover, an upper-level ontology is adopted, which allows to model generic concepts that can be specialized for the proposed ontology, as well as fosters better interoperability and reusability of the human activity model.

5.1.1. Upper-level ontology

The DOLECE Ultralite ontology (DUL, http://ontologydesignpatterns.org/wiki/Ontology:DOLCE+DnS_Ultralite) is chosen as an upper ontology, since it is a broadly reused and comprehensive foundational ontology, moreover, many ontology-based frameworks have been built on it or aligned with it. The DUL ontology provides a set of upper level concepts to foster interoperability among many middle and lower level ontologies. The highest concept of the DUL ontology is dul:Entity, under the umbrella of this concept, four disjoint classes are defined: dul:Event, dul:Object, dul:Abstract and dul:Quality. The class dul:Event stands for perduring entities that unfold over time (e.g. the event takes up time). The class dul:Object represents enduring entities that unfold over space (e.g. living things as well as non-living things). The class dul:Quality stands for the characteristic of the instances of dul:Event and dul:Object. The class dul:Abstract represents value spaces (e.g. the time interval of an event).

5.1.2. Ontology statements

For the user modelling, the Friend of a Friend (FOAF, http://xmlns.com/foaf/spec/) ontology enables to model generic information about a user, such as the name, age and profile. However, it does not support the role and user preference modelling. Hence, the vocabulary of Semantically Interlinked Online Communities core ontology (SIOC, http://www.w3.org/Submission/sioc-spec/) is reused to model the role that the user plays. The RECommendation Ontology (RECO, http://purl.org/ontology/rec/core) is a vocabulary to express user preference, it can be reused to model the user’s preference of an activity. For sensor modelling, the Sensor and Sensor Network (SSN, https://purl-oclc-org.web.bisu.edu.cn/NET/ssnx/ssn) ontology is reused to model sensors deployed in smart home, it is created by W3C Semantic Sensor Network Incubator Group (SSN-XG, http://www.w3.org/2005/Incubator/ssn/wiki), which associates sensing devices with observations and measurements. The SWRL Temporal ontology (http://protege.cim3.net/cgi-bin/wiki.pl?SWRLTemporalOntology) is reused to model temporal context, it defines a set of built-ins that can be used in SWRL rules to represent temporal information. The place ontology design pattern (ODP, http://ontologydesignpatterns.org/wiki/Submissions:Place) and the vocabulary inherited from the DUL ontology are reused to model spatial context in home areas. The Simple Event Model (SEM) [29] is reused to represent domain activity concepts and the Mereology pattern of Event Model F [22] is reused to model the activity state as well as relative temporal relations between activities.

Fig. 1.

The conceptual modules of the human activity model.

Figure 1 describes the conceptual modules of the human activity model. This model consists of a set of modules, which can be classified into application ontologies, domain ontologies and general ontologies, based on the subject of their conceptualization identified in [10]. Application ontologies are built for the certain application. Here, the human activity ontology is extended with possible extensions for the certain application scenarios. Domain ontologies model common concepts and aspects for the certain domain, which could be reused for more specific scenarios. In this case, domain activity ontology, physical object ontology, sensor ontology and ADL pattern are classified in this category. General ontologies are universal and domain-independent areas that can be reused across various domains of knowledge. In this sense, temporal ontology, place ontology and user ontology are classified in this category. It is worth to mention that, the more general ontology is, the fewer possibilities exist for usability but more possibilities exist for reusability among heterogeneous applications.

5.2. Core concepts

The concept of ha:Entity is defined as the top-level concept of the human activity ontology. It consists of the following core concepts: ha:User is used to model the users, their roles and preferences; ha:Sensor is used to model sensor devices and their measurements; ha:Place is used to represent the house areas; ha:PhysicalObject is used to represent the household objects; ha:Temporal is used to model the temporal information; ha:Activity is used to represent the activities monitored in smart home; ha:ADLSituation, ha:ADLDescription and ha:Concept are used to model the ADL pattern of a user. The detailed description is presented in this section.

5.2.1. User

The user ontology is proposed to model the information of intended users related to smart home applications. The function of the user ontology is two-fold: (1) to describe the static information (e.g. name and age) of an individual and identify the role that the individual plays (e.g. Maria plays activity actor role). (2) to present the profiles and activity preferences of the individual (e.g. Maria’s heart rate is recorded as her profiles and she prefers watch TV in the dining room with light off). Due to the roles, profiles and activity preferences are completely different for different individuals, it is necessary for the user ontology to capture this complexity inherent to individuals and their involvements in smart home applications.

The concept of ha:User is modelled to be equivalent to foaf:Agent, sioc:User and reco:Agent. It is aligned with the DUL ontology as an equivalent class of dul:Agent. The vocabulary of the FOAF ontology is reused to represent the users’ static information and their profiles. The individual who involved in smart home applications is modelled as an instance of foaf:Person. The properties of foaf:name, foaf:age and foaf:gender are used to express a user’s name, age and gender, respectively. The user profile is described by the ha:Profile class (equivalent to foaf:PersonalProfileDocument) and represents the user in different contexts, where context is widely interpreted. The relations between ha:Profile and foaf:Person are established by means of the property ha:profiles. The role that a foaf:Person plays, is modelled as an instance of the sioc:Role class to represent a collection of the individuals, the corresponding property ha:plays links up the instances of foaf:Person and sioc:Role. The vocabulary of RECO ontology is added to represent the preferences of a particular user. The perception of activities occurring in the real scenarios heavily relies on the point view of the actor. The particular way that an actor prefers to perform an activity, is defined by reco:Preference, and linked to foaf:Person via the ha:hasPreference property.

5.2.2. Sensor

The input of smart home system is a stream of time-stamped values produced by heterogonous sensors, which deployed in the environments or attached on the users to observe a significant variables. In smart homes, data collected from different sensors are heterogeneous in data formats. For example, a contact sensor provides a two-state binary value while an accelerometer offers 3-D coordinates. The lack of formalized representation of these data would lead to weak interoperability and reusability within different systems. The basic idea of sensor measurements modelling is annotating heterogonous sensor data with formalized semantic, which could provide a mechanism to manage the perceived sensor data by creating semantic content in order to increase interoperability as well as provide contextual information.

The conceptualizations of sensor measurements module relating to sensing and observation are reused from the SSN ontology. The SSN is a remarkable work for the formal representation of sensor capabilities, properties, observation and measurements process through high-level specifications. This ontology is developed following the principle of minimal ontological commitments, it introduces a minimal set of classes and relations entered around the notions of ssn:Sensor and ssn:Observation to make it reusable for various application domains. The ssn:Sensor class is defined as an entity that observes and transforms incoming stimuli into another, often digital, representation though the object property ssn:detects. The ssn:Observation of sensor is defined as a subclass of dul:Situation, which is a social object that satisfies some descriptions. It places related classes, including sensor (ssn:observedBy), property (ssn:observedProperty) and observation value (ssn:hasValue) in an interpretative context.

One limitation of existing SSN ontology is that, the SSN ontology cannot support the association among context entities. In order to overcome this limitation, the new properties are defined to extend the sensor associated with feature of interest and property. The ha:locatedIn property is used to assert that a sensor is deployed in a house area. The ha:attachedTo property is used to associate the sensor to the user or physical object.

5.2.3. Place

The place in smart home, such as dining room, bathroom, kitchen, and bedroom, could be used to characterize the location of a sensor, user or situation. In this sense, the place ontology is proposed based on the place ontology design pattern to model knowledge about generic locations within smart home. The Place ontology design pattern is implemented in DUL and the relevant DUL statements, a dul:Place is something that has at least one dul:isLocationOf property (its inverse property is ha:hasPlace). Furthermore, The place ontology can be linked with SSN ontology though ha:hasObservedPlace, which defined as a subclass of ssn:hasProperty, to express the observed location of sensor.

5.2.4. Physical object

The physical object ontology reuses the concept of object module of the DUL ontology to describe the knowledge of household objects that involved in activity, for instance, the activity “take medicine” occurs with the usage of medicine box. Moreover, the vocabulary extracted from [15] is also reused to classify the household objects into two categories: fixed and moveable. The fixed physical objects include fixtures and fittings (e.g. bed, door, stove) and the moveable ones refer the physical objects that have changeable locations (e.g. telephone, chair), such category can facilitate to keep track of an entity. The instance of dul:PhysicalObject class can link the place ontology through the dul:hasLocation property to express the location of the household object (e.g. the stove is located in the kitchen). In addition, the property ha:attachedTo (its inverse property is ha:hasSensor) is used to assert that a sensor is attached to a household object (e.g. a RFID sensor is attached to a cup).

5.2.5. Temporal

The SWRL temporal ontology is reused as a principled temporal model to enforce a consistent representation of temporal content and properties. This ontology provides a vocabulary for describing complex interval-based temporal information, as well as the formal underpinning with a library of SWRL built-ins to perform temporal operations.

Fig. 2.

The concepts of SWRL temporal ontology.

As shown in Fig. 2, an activity holds the valid-time is represented by using the class swrlto:ValidTime (equivalent to dul:TimeInterval) via the property swrlto:hasValidTime. The swrlto:ValidInstant and swrlto:ValidPeriod are donated as two subclasses of swrlto:ValidTime to describe the instants and intervals respectively. The instance of swrlto:ValidPeriod is the thing with extent, it links xsd:Date Time via two properties: swrlto:hasStart and swrlto:hasFinish. The instance of swrlto:ValidInstant is, intuitively, point-like in that it has no interior points, it links xsd:Date Time via ha:hasInstant property (equivalent to swrlto:hasTime). Generally, an instant can be treated as an interval where the start time and finish time are the same. The granularity of time (e.g. year, month, day, hour, minute, etc.) is represented by using swrlto:Granularity class (a subclass of dul:Unit of measure) to specify the accuracy of the temporal information. Durations are represented by the class swrlto:Duration, which has a functional integer property swrlto:hasCount and a functional granularity property with a range of swrlto:Granulairty.

The SWRL temporal ontology can be linked with ssn:Observation of SSN ontology though ssn:observationSamplingTime and ssn:observationResultTime to express the information about observation times.

5.2.6. Activity

ADL is specified with related contextual information, e.g. occurring time, place, related object. A domain ontology serves as a core vocabulary for the basic activity contextual information representation. The Simple Event Model (SEM) is chosen as the basis of the ADL domain ontology. The SEM is built on DOLCE+DnS Ultralite ontology. It takes main aspects (e.g. actor, place, time) of an activity into consideration and allows the user to integrate external ontologies, such as temporal ontology, for further description of an activity.

Fig. 3.

Conceptual ADL domain ontology.

The proposed activity domain ontology (depicted in Fig. 3) is constructed as a core vocabulary, which enables to drive the derivation of composite activities from the contextual information. Based on SEM, the ADL is represented as an instance of the ha:Activity class (equivalent to sem:Event), an ha:Activity instance is matched with one or more instances of the context entities in order to present a specific activity. The user who performs the activity is defined as the individual of the sem:actor class by using sem:hasActor property. The place where an activity occurs is represented by using sem:hasPlace property. The corresponding sematic of composite activities correlation can be represented by the use of ha:hasSubActivity (equivalent to sem:hasSubEvent) property. Additionally, the concept ha:PhysicalObject is added to the SEM ontology to represent the relevant household object, the association between one ha:PhysicalObject and one ha:Activity is implemented by using the ha:hasPhysicalObject property (e.g. the household object “broom” is related to the activity “sweeping the floor”). The ha:hasConstraint is also added to specify the activity with ssn:Property. For example, the human posture can be viewed as a sensor property to specify the activity that has an ambiguous problem. Considering the situation that the pressure sensor attached to the bed in the bedroom is triggered, there are two possible activities: sleeping in the bed or sitting on the bed to read a newspaper, the addition of posture “lying down” or “sitting” can verify the happening activity in a more accurate manner.

5.2.7. ADL pattern

The functional requirements on human activity ontology are represented by specialized instantiations of the description and situation (DnS) ontology pattern of the DUL ontology. DnS pattern enables formally precise representations of various contextualized views on the instances of the dul:Event class, which can be used to describe the n-ary relations among multiple individuals of dul:Event and dul:Object. Based on DnS, the proposed ADL pattern implements the relation of situations and descriptions with individuals of the ha:Activity (a subclass of dul:Event) and ha:ActivityType (a subclass of dul:Concept) classes. As shown in Fig. 4, The monitoring result is expressed by an ha:ADLSituation that dul:satisfies an ha:ADLDescription. The classes of ha: ADLSituation and ha:ADLDescription are aligned with the DUL ontology to be equivalent classes of dul:Situation and dul:Description, respectively. The ha:ADLSituation includes context information related to the activity being described. The ha:ADLDescription serves as the corresponding deceptive context on a set of entities to create views on an ha:ADLSituation. The concept of ha:ActivityType classifies the ha:Activity class and describes the interpretation of an activity in ha:ADLSituation.

Some vocabularies in the mereology pattern of Event Model F are reused to construct an extension of the ADL pattern to express temporal relationships among activities. The mereology pattern considers the interrelationships as composition of activities and enables to express their temporal relations by using temporal inference.

As shown in Fig. 4, the instances of ha:Activity from the domain activity ontology is included in F:CompositionSituation, which satisfies a F:CompositionDescription. The F:Composite and F:Component are defined as the subclasses of ha:ActivityType to express the composites activity and its constitutive activities. Activities that play F:Component roles could be further qualified by the temporal constraints to describe the temporal relations among these activities, such temporal constraint are formally expressed by the instances of the F:TemporalConstraint (a subclass of F:CompositionConstraint) that is defined by The F:CompositionDescription.

Fig. 4.

Conceptual ADL pattern.

Allen’s Time Calculus can be integrated into ADL pattern to support the temporal inference. Table 1 shows the detailed Allen’s temporal relations of activities [1], which position any pair of intervals by comparing their starting and ending time points to support qualitative temporal knowledge representation. In terms of the temporal intervals, the activities and their relations can be modelled as sequential activities, concurrent activities and interleaved activities.

Sequential activities: Sequential activities refer one activity is performed either before or after another activity. The before/after and meets/metBy of Allen relations are used to associate sequential activities with their temporal intervals. It is worth to mention that the before/after relation has a time length between two intervals of activities, and the meets/metBy relation could be views as a specification of the before/after relation without a zero time length between two intervals.

Concurrent activity: Concurrent activities occur at the same time, thus they share the same time intervals fully or partially. There are nine relations used to model concurrent activities: overlaps/overlappedBy, starts/startedBy, during/contains, finishes/finishedBy and equals. The ovelaps/overlappedBy relation is used to model activities that have partially shared temporal intervals. But inherently, the start time point of one interval is before the other one. The starts/startedBy relation indicates that two intervals have the same start time but one has a longer interval length. Conversely, the finishes/finishedBy relation refers the same finish time of two intervals. The during/contains relation shows that the interval of a composite activity encloses the interval of its composing activity. The equals relation is only used when two activities take place parallelism in the same interval, and their intervals start and finish at the same time point.

Interleaved activities: have time intervals that “preempt” each other, which indicates that an activity may occur within a long and complex activity, the longer interval contains the shorter one. The during/contains relation is also used to describe the interleaved activities, but different from concurrent activities, the activity with the shorter interval is not a component of the other.

Table 1

Thirteen temporal relations (x and y are temporal intervals)

6. Implementation and evaluation

6.1. Formalization

The proposed human activity ontology is constructed by using Protégé ontology editor and represented in OWL2. OWL2 is a semantic web ontology language based on DL. It contains a set of constructors for specifying conceptual structures and relationships (e.g. equivalent, disjoint and subsumption) as well as supporting inference and reasoning. Figure 5 presents a hierarchical fragment of key classes within human activity ontology.

Fig. 5.

A snapshot of classes in human activity ontology.

This human activity ontology enables to model knowledge at two levels of abstraction. One is obtaining raw data and generating related contextual information from context entities. The other is deducing the high-level activity-concerned information from the raw data and contextual information. The sensor devices that deployed in smart home environments capture the detailed signals of an activity when triggered. These data are received and stored for further processing. Moreover, the ontology enriches these signals by adding semantic metadata to the raw sensor data. Then the semantic contexts as well as sensor data are potentially generated to gain the high-level context for smart home applications.

Due to the inability of OWL to express complex rule formations, the Semantic Web Rule Language (SWRL) is used to add rule capabilities to OWL-based ontologies. The SWRL-based rules are made up of an antecedent and a consequent [13], these rules can be used together with OWL constructors to ensure the decidability, soundness and completeness in reasoning. After the SWRL rules are created, they could be used for inference and reasoning by using a reasoning tool (e.g. Pellet, https://github.com/complexible/pellet). Here, three example rules by using the syntax of SWRL temporal ontology built-ins are described as follows.

Example rule 1.

hasValidPeriod ( $? X$ , $? pX$ ) ∧ hasValidPeriod ( $? Y$ , $? pY$ ) ∧ hasFinish ( $? pX$ , $? fX$ ) ∧ hasStart ( $? pY$ , $? sY$ ) ∧ before ( $? fX$ , $? sY$ ) → before ( $? pX$ , $? pY$ )

Example rule 1 states that, if Activity X has an interval $pX$ and activity Y has an interval $pY$ , the finish time of $pX$ is before the start time of $pY$ , then the fact “ $pX$ is before $pY$ ” is inferred according to composite activity analysis on Allen’s Time Calculus, it is inferred that activity X and activity Y take place sequentially and activity X takes place before activity Y.

Example rule 2.

hasValidPeriod ( $? X$ , $? pX$ ) ∧ hasValidPeriod ( $? Y$ , $? pY$ ) ∧ hasValidPeriod ( $? Z$ , $? p Z$ ) ∧ before ( $? pX$ , $? pY$ ) ∧ before ( $? pY$ , $? p Z$ ) → before ( $? pX$ , $? pY$ )

Example rule 2 states that, if the interval of X is before the interval of Y, and the interval of Y is before the interval of Z, then the fact of “the interval of X is before the interval of Z” is inferred.

Example rule 3.

hasValidPeriod ( $? X$ , $? pX$ ) ∧ hasValidPeriod ( $? Y$ , $? pY$ ) ∧ hasStart ( $? pX$ , $? s X$ ) ∧ hasFinish ( $? pX$ , $? fX$ ) ∧ hasStart ( $? pY$ , $? sY$ ) ∧ hasFinish ( $? pY$ , $? f Y$ ) ∧ before ( $? s X$ , $? sY$ ) ∧ before ( $? f Y$ , $? fX$ ) → during ( $? pY$ , $? pX$ ), isComponentFor( $? Y$ , $? X$ )

Example rule 3 derives the during interval relation by using the start and finish points of two given activities. It asserts the activity Y that with shorter time interval is a component activity of activity Y which has a longer time interval.

6.2. Scenario with ontology extensions

To evaluate the proposed human activity ontology, this section describes a proof of concept in smart home domain. The specified scenario presented in Section 3 is revisited to provide an overview on how to represent the user and activity information with concepts and properties.

In terms of user aspect, Maria, Carlos and Juan are created as individuals of the foaf:Person class, and play (via the property ha:plays) the roles of actor, health expert and caregiver, respectively. In terms of SH context aspect, the accelerometer, ECG, RFID, motion sensor, pressure sensor, temperature sensor and light sensor are created as individuals of the ssn:Sensor class. Bedroom is created as the individual of the ha:Place class. Bed, medicine box and cup are created as individuals of the ha:PhysicalObject class. As concerned activities, “sleep”, “take medicine” and “use cup” are created as individuals of the ha:activity class. Take the activity “sleep” for example, it is defined as “an Actor: Maria uses bed in bedroom for 8 hours”, this description can be expressed with the following OWL axioms:

Sleep ≡ Activity ⊓ hasActor(Maria) ⊓ hasPhysicalObject(Bed) ⊓ hasPlace(Bedroom) ⊓ hasDuration(8[int]hour)

Additionally, in the scenario, two health features (heart rate and posture) and two environment features (temperature and light) are also monitored to help understand Maria’s state when she performs the activity in home areas. As shown in Fig. 6, two modules: PhysiologicalState and EnvironmentalState, are added as the extensions of human activity ontology to provide additional health and environment information in terms of a specific type of property. The concept of PhysiologicalState consists of two subclasses: HeartRate and Posture. The concept of EnvironmentalState consists of two subclasses: Temperature and Light. These two modules are added to be the subclasses of ssn:Property. The values of which are the observation of the corresponding sensors (in this case, Electrocardiogram sensor (ECG) and Accelerometer for health information, temperature sensor and light sensor for environment information).

Fig. 6.

Extended modules for features related to PhysiologicalState and EnvironmentalState.

7. Conclusion and future work

This paper has presented the engineering of human activity ontology to support the phase tasks of the development process in user, SH context and activity modelling within heterogeneous smart home applications. The development process of the proposed ontology follows the NeON methodology, towards the definition of a set of reused and shared representational primitives to model the user and activity knowledge in smart home domain. Based on the scenario specification and requirements gathered from the current research literature, the main methodological decision is to reuse widely accepted conceptualizations in existing ontologies, partly integrate the needed conceptualizations, as well as extend them with new concepts and properties to satisfy the full requirements. In addition to the description of the concepts and properties of the human activity ontology, the paper also presents the implementation by using OWL and the evaluation based on the motivated scenario to do the proof-of-concept.

Contrasted to developing smart home applications for very particular configurations, the aim of the proposed model is to facilitate the generic application development in spite of different aspects such as diverse sensor devices or different target activities. Moreover, the adoption of the DOLCE+DnS Ultralite (DUL) ontology as the upper ontology facilitates the integration of the conceptualization from other ontologies, and achieves a high degree of reusability and interoperability among heterogeneous smart home applications in an open and extensible manner.

The future work aims to extend current activity ontological model for capturing more complex activities with additional rules and exploring effective algorithms for activity recognition. Moreover, there exist sensor failure and reasoning incapability, thus, handling uncertainty knowledge in sensor data will be another direction of the future work.

Footnotes

Acknowledgements

Authors would like to thank the Telematics for Knowledge and Information Society research group (T>SIC), of Universidad Politécnica de Madrid, for its support to this research. Authors also acknowledge the China Scholarship Council which has partially supported the research described in this paper.

References

[1]

J.F.

Allen, Maintaining knowledge about temporal intervals, Communications of the ACM26 (1983), 832–843.

[2]

H.T.

Chaminda,

Klyuev and

Naruse, A smart reminder system for complex human activities, in: 14th IEEE International Conference on Advanced Communication Technology, 2012, pp. 235–240.

[3]

Chen,

Finin and

Joshi, The SOUPA ontology for pervasive computing, in: Ontologies for Agents: Theory and Experiences, Springer, 2005, pp. 233–258.

[4]

Chen,

Nugent and

Bashrawi, Semantic data management for situation-aware assistance in ambient assisted living, in: 11th ACM International Conference on Information Integration and Web-Based Applications & Services, 2009, pp. 298–305.

[5]

Chen,

Nugent and

Wang, A knowledge-driven approach to activity recognition in smart homes, IEEE Trans. Knowledge and Data Engineering24 (2012), 961–974.

[6]

Daoutis,

Coradeschi and

Loutfi, Grounding commonsense knowledge in intelligent systems, Journal of Ambient Intelligence and Smart Environments1 (2009), 311–321.

[7]

A.K.

Dey, Understanding and using context, Personal and Ubiquitous Computing5 (2001), 4–7.

[8]

Gangemi and

Mika, Understanding the semantic web through descriptions and situations, in: The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE, Springer, Berlin, Heidelberg, 2003, pp. 689–706.

[9]

Giannakouris, Ageing characterises the demographic perspectives of the European societies, Statistics in Focus72 (2008).

10.

[10]

Gomez-Perez,

Fernández-López and

Corcho-Garcia, Ontological engineering, Computing Reviews45 (2004), 478–479.

11.

[11]

H.J.T.

Horst and

A.G.

Sinitsyn, Structuring reasoning for interpretation of sensor data in home-based health and well-being monitoring applications, Journal of Ambient Intelligence and Smart Environments4 (2012), 461–476.

12.

[12]

Hussain,

Ali,

W.A.

Khan,

Afzal,

Lee and

Latif, Recommendations service for chronic disease patient in multimodel sensors home environment, Telemedicine and e-Health21 (2015), 185–199.

13.

[13]

Keßler,

Raubal and

Wosniok, Semantic rules for context-aware geographical information retrieval, in: Smart Sensing and Context, Springer, Berlin, Heidelberg, 2009, pp. 77–92.

14.

[14]

Lloret,

Canovas,

Sendra and

Parra, A smart communication architecture for ambient assisted living, IEEE Communications Magazine53 (2015), 26–33.

15.

[15]

L.M.

McAvoy,

Chen and

Donnelly, A generic upper-level ontological model for context-aware applications within smart environments, in: Ubiquitous Computing and Ambient Intelligence. Context-Awareness and Context-Driven Interaction, Springer International Publishing, 2013, pp. 270–277.

16.

[16]

Meditskos,

Dasiopoulou and

Kompatsiaris, Metaq: A knowledge-driven framework for context-aware activity recognition combining SPARQL and OWL 2 activity patterns, Pervasive and Mobile Computing (2015).

17.

[17]

Nazerfard and

D.J.

Cook, Using Bayesian networks for daily activity prediction, in: AAAI Workshop: Plan, Activity, and Intent Recognition, 2013.

18.

[18]

Okeyo,

Chen and

Wang, Combining ontological and temporal formalisms for composite activity modelling and recognition in smart homes, Future Generation Computer Systems39 (2014), 29–43.

19.

[19]

Pulkkinen,

Y.S.

Son,

Lee,

Y.H.

Lee,

Sallinen and

J.H.

Park, Progressive monitoring and treatment planning of diabetes mellitus in smart home environment, in: IEEE International Conference on Consumer Electronics, 2013, pp. 206–207.

20.

[20]

Riboni,

Bettini,

Civitarese,

Z.H.

Janjua and

Helaoui, Extended report: Fine-grained recognition of abnormal behaviors for early detection of mild cognitive impairment, arXiv:1501.05581, 2015.

21.

[21]

L.Á.

San Martín,

Peláez and

González, Environmental user preference learning for smart homes, Intelligent Environments2 (2009), 177–184.

22.

[22]

Scherp,

Franz,

Saathoff and

Staab, F-a model of events based on the foundational ontology dolce+DnS ultralight, in: Fifth ACM International Conference on Knowledge Capture, 2009, pp. 137–144.

23.

[23]

K.L.

Skillen,

Chen,

Nugent,

Donnelly,

Burns and

Solheim, Using SWRL and ontological reasoning for the personalization of context-aware assistive services, in: 6th ACM International Conference on PErvasive Technologies Related to Assistive Environments, 2013, p. 48.

24.

[24]

M.C.

Suárez-Figueroa, NeOn methodology for building ontology networks: Specification, scheduling and reuse, Ph.D. dissertation, Universidad Politecnica de Madrid, 2010.

25.

[25]

S.P.

Syväniemi,

Purhonen and

Ovaska, Situation-based and self-adaptive applications for the smart environment, Journal of Ambient Intelligence and Smart Environments4 (2012), 491–516.

26.

[26]

Tong and

Chen, Latent-dynamic conditional random fields for recognizing activities in smart homes, Journal of Ambient Intelligence and Smart Environments6 (2014), 39–55.

27.

[27]

M.Z.

Uddin,

D.H.

Kim,

J.T.

Kim and

T.S.

Kim, An indoor human activity recognition system for smart home using local binary pattern features with hidden Markov models, Indoor and Built Environment22 (2013), 289–298.

28.

[28]

C.G.

Vallerand,

Abdulrazak and

Giroux, A context-aware service provision system for smart environments based on the user interaction modalities, Journal of Ambient Intelligence and Smart Environments5 (2013), 47–64.

29.

[29]

W.R.

Van Hage,

Malaisé,

Segers,

Hollink and

Schreiber, Design and use of the simple event model (SEM), Web Semantics: Science, Services and Agents on the World Wide Web9 (2011), 128–136.

30.

[30]

X.H.

Wang,

D.Q.

Zhang,

Gu and

H.K.

Pung, Ontology based context modeling and reasoning using owl, in: Second IEEE Annual Conference on Pervasive Computing and Communications Workshops, 2004, pp. 18–22.

31.

[31]

Zhang,

McCullagh,

Nugent,

Zheng and

Black, An ontological framework for activity monitoring and reminder reasoning in an assisted environment, Journal of Ambient Intelligence and Humanized Computing4 (2013), 157–168.