Shared learning activity labels across heterogeneous datasets

Abstract

Nowadays, the advancement of sensing and communication technologies has led to the possibility of collecting a large amount of sensor data, however, to build a reliable computational model and accurately recognise human activities we still need the annotations on sensor data. Acquiring high-quality, detailed, continuous annotations is a challenging task. In this paper, we explore the solution space on sharing annotated activities across different datasets in order to enhance the recognition accuracies. The main challenge is to resolve heterogeneity in feature and activity space between datasets; that is, each dataset can have a different number of sensors in heterogeneous sensing technologies and deployed in diverse environments and record various activities on different users. To address the challenge, we have designed and developed sharing data and sharing classifiers algorithms that feature the knowledge model to enable computationally-efficient feature space remapping and uncertainty reasoning to enable effective classifier fusion. We have validated the algorithms on three third-party real-world datasets and demonstrated their effectiveness in recognising activities only with annotations from as little as 0.1% of each dataset.

Keywords

Activity recognition uncertainty reasoning feature space remapping ontologies smart home transfer learning

1. Introduction

Sensor-based human activity recognition (HAR) has been playing a significant role in many applications, which enables to provide customised services to suit people’s current context. Various machine learning including recent deep learning techniques have been applied to HAR in feature extraction [2] and sensor fusion [16] and achieved promising accuracy on recognising daily activities. However, these techniques heavily rely on labelled training data in order to build a robust computational model.

Labelling sensor data with activities is a time-consuming and cost-sensitive task. Reducing the reliance on training data has long been a challenging research topic, and different approaches have been attempted, including unsupervised learning [32], active learning [1], co-training [6], and transfer learning [27]. The majority of these approaches have reduced the annotation to a certain degree, but the annotation burden on individual users is still heavy, and does not scale to a large number of users.

To directly tackle this challenge, we propose a SLearn approach to relieve the annotation burden on individual users via cross learning on scarce and partially annotated data from multiple users to achieve satisfactory activity recognition accuracies. The hypothesis is that as long as each user contributes a small number of labelled examples (even though these examples might not cover a complete set of activity types), SLearn will learn annotated examples across all the users and be able to build an activity recognition model for each user to recognise all the activities. For example, if a user A’s data only contains annotations on ‘sleeping’ and a user B’s only contains annotations on ‘preparing meal’, SLearn will take both users’ data and build activity models to be able to recognise these two activities on both users. In this way, SLearn reduces labelling on individual users and still builds a robust model to recognise activities.

Fig. 1.

Three smart home deployments observing the same set of activities but with different spatial layouts and a different number of sensors. Each table shows the averaged activation ratios of sensors (e.g., S1 or S2, mapping to red dots in each house setting) per each activity class (e.g., ‘Leave home’ or ‘Sleep’).

The challenge faced by SLearn is the heterogeneity in different datasets. Take Fig. 1 as an example where each smart home has different spatial layouts and is deployed with a different number of sensors with different sensing technologies. Transferring activity models across these smart homes, especially, mapping and/or aligning features extracted from these sensors, is a daunting task, as their feature space can be completely disparate. Here we focus on binary sensors, including RFID sensors, door sensors, and pressure sensors. Transfer learning across datasets on binary sensors is currently under-explored. Most transfer learning techniques on sensor-based activity recognition focus on accelerometer data that have the uniform format of sensor data [27].

We have explored preliminary sharing data and sharing classifiers algorithms to integrate knowledge- and data-driven approaches in our earlier work [30]. The algorithms have achieved promising results in gaining reasonably good accuracy with a small amount of training data from each dataset. However, the algorithms struggle when there exists high heterogeneity between datasets and sensor data are noisy, making it uncertain to select appropriate activity labels. To tackle this challenge and improve the accuracy of shared learning, we use the recent advance in uncertainty reasoning to support robust classifier fusion in the face of conflict and uncertainty among classifiers in different datasets. We design a new hybrid approach to combine sharing data and classifiers to complement their limitations and reduce the risk of negative transfer. We evaluate the algorithms on three third-party datasets and demonstrate their effectiveness by comparing them against the state-of-the-art classification techniques.

The rest of the paper is organised as follows. Section 5 reviews the mainstream work on addressing the scarcity of labelled data and identifies the difference between them and SLearn. Section 2 describes the SLearn approach and Section 3 introduces the evaluation methodology and experiment setup. Section 4 performs an evaluation and discusses the strength and limitation of SLearn Algorithms, and Section 6 concludes with some suggested future work.

2. Proposed approach

We hypothesise that by shared learning across different datasets, we can build an activity model that can recognise these users’ activities accurately only with a small amount of training data. Let $A = {a_{1}, a_{2}, \dots, a_{k}}$ be a set of k activities of interest, $DS = {D_{1}, D_{2}, \dots, D_{n}}$ be a collection of n datasets, where $D_{i}$ ( $1 ⩽ i ⩽ n$ ) consists of $p_{i}$ partially labelled training instances $L_{i} = {(x_{j}, y_{j})}_{j = 1}^{p_{i}}$ ( $y_{j} \in A_{i}$ ) and $q_{i}$ unlabelled instances $U_{i} = {x_{j}}_{j = 1}^{q_{i}}$ . $A_{i}$ is the set of activity classes being annotated on $D_{i}$ and $A_{i} \subseteq A$ . SLearn aims to combine the training instances $L_{i}$ from $DS$ to build an activity model to recognise all the activities in A on unlabelled instances $U_{i}$ in $DS$ . A detailed definition of SLearn is given in our previous work [30].

In the following, we will explore the solution space to address this question. We will look into two directions: sharing classifiers and sharing training data. For both approaches, the key enabler is feature space remapping; that is, transforming sensor feature from one dataset to another.

2.1. Feature space remapping

Different feature remapping strategies have been proposed [18] and a promising approach is a meta-feature based mapping function for event-driven sensors in smart home environments [5]. The authors have defined a range of meta-features about each sensor; for example, the average sensor event frequency over 1-hour time periods, over 3/8/24-hour periods, the mean and standard deviation of the time between this sensor event and the next sensor event, and the probability of the next event is from the same sensor. These meta-features are used as a heuristic to guide the mapping process. This is a data-driven approach for feature space remapping, but its performance is affected by the activity routine of various users. For example, one user might often have breakfast at 6am while the other might have at 9am, or one user prefers having shower before breakfast while the other prefers the other way around so that the sensors might be mis-matched on the time scale. In addition, the density of the sensor deployment and the frequencies and sensitivity of sensors reporting events affect the mapping on the intervals between events. For example, one environment can be more densely deployed with sensors so that the time distances between events reported by different sensors can be significantly shorter than the other environment set up with much fewer sensors.

To reduce the impact of such differences in each dataset, we adopt a more general approach – semantics-based feature mapping. We will use the common knowledge, which has demonstrated generality across different smart home datasets [32]. The principle of semantics-based feature mapping is to compute similarity between a pair of sensors based on where they are deployed and which object they have attached to. Both location and object concepts are organised in an ontological hierarchy, from which a conceptual similarity measure [28] is applied to calculate the distance between two concepts. For example, a sensor in House A can be described with Kitchen (i.e., a location concept) and Cup (i.e., an object concept), and a sensor in House B with Kitchen and Pan. If the conceptual similarity between Cup and Pan is 0.22, then we can calculate the similarity between these two sensors as 0.61 ( $= (1 + 0.22) / 2$ ). We construct a simple location ontology based on their functionality; e.g., Kitchen ⊑ CookingArea ⊑ DiningArea, and an object ontology based on WordNet [15]; e.g., Cup ⊑ Crockery ⊑ Tableware. The semantic similarity between ontological concepts is based on the hierarchy [28]. A detailed description of the ontologies and semantic similarities between sensors can be found [33].

Definition 1.
Given an instance $X_{I} = [x_{I, 1}, x_{I, 2}, \dots, x_{I, n_{I}}]$ , a semantics-based feature mapping function is defined as $θ (X_{I}) = X_{II}$ , denoted as $[x_{II, 1}, x_{II, 2}, \dots, x_{II, n_{II}}]$ , where $\forall 1 ⩽ j ⩽ n_{II}$ , $x_{II, j} = \sum_{i \in S_{j}} x_{i} * {sim}_{S} (s_{I, i}, s_{II, j}) / | S_{j} |$ , $S_{j}$ is a collection of sensors in the feature space I that are similar to the sensor j in II. That is, $S_{j} = {s_{l} | 1 ⩽ l ⩽ n_{I}, {sim}_{S} (s_{I, l}, s_{II, j}) > ε}$ , where ε is the threshold to choose similar sensors and ${sim}_{S}$ is the similarity between two sensors.

Using Definition 1, we can perform sensor feature remapping. That is, a value $x_{II, j}$ in a converted instance $X_{II} \in χ_{II}$ is the weighted average of the probabilities of all the similar sensors in the source feature space $χ_{I}$ to the jth sensor in $χ_{II}$ and the weight is their sensor similarity. In another word, we try to estimate the probability of the jth sensor reporting events by looking at the probabilities of all of its similar sensors in the source dataset.

Figure 2 illustrates an example of the above process. Assume that there are two simplified datasets I and $II$ , each having 2 and 3 sensors respectively, and their similarity scores have been calculated based on the similarity of their attached objects and deployed locations. Given a current sensor feature $X_{I}$ 1
¹
We extract sensor feature from sensor events as $X = [x_{1}, \dots, x_{n}]$ , where n is the number of sensors being deployed and $x_{i}$ indicates the probability of the ith sensor reporting an event during a certain time interval (e.g., every 30 or 60 seconds); that is, $x_{i} = N_{i} / N$ , where $N_{i}$ is the number of events reported by the ith sensor and N is the total number of events being reported in an interval. This is a common approach for extracting features on binary event-driven sensors [6,26,33].

in the dataset I, we need to simulate a sensor feature $X_{II}$ in the other dataset. First, for each sensor in $II$ , we need to identify similar sensors in I. Assume that the similarity threshold is 0.5, according to the formula in Definition 1, we identify the similar sensor sets $S_{j}$ ( $j = 1, 2, 3$ ) for each sensor in the dataset $II$ as ${s_{I, 1}}$ , ${s_{I, 1}}$ , and ${s_{I, 2}}$ . Then the probability on each sensor is the averaged contribution from their similar sensors.

Fig. 2.
An example of sensor feature space remapping.

Algorithm 1
SC: Share classifiers
2.2. Sharing classifiers

To share classifiers, we design a uncertainty-driven algorithm; that is, when the classifier from the current dataset cannot confidently infer an activity label to a given example, then we acquire labels from the classifiers from the other datasets. This is inspired by active learning; that is, identify uncertain examples and query human operators for annotation. The difference here is that we do not query human operators, but classifiers in the other datasets.

We describe the process in Algorithm SC (Algorithm 1). To make it work, we will need to solve two problems: (1) how to evaluate whether an example is uncertain to label, and (2) how to integrate results from multiple classifiers. To address the first question, we will employ the most common uncertainty sampling strategies from active learning, which are least confidence, margin sampling, and entropy [21]. We have run experiments on these three strategies and in the end settle with information entropy strategy that yields the best accuracy. With the uncertainty sampling strategies, we can determine which example is not certain to be labelled from its current classifier.

Then we will perform a feature space remapping that converts it to examples in the other datasets to allow the other classifiers to label. Once the other classifiers complete the inference, we will need to collect their results and integrate them to make a final decision. The existing approaches to combine classifiers include ensemble methods like boosting, bagging, and stacking. However, we are constrained by the size of training data, so a training data craving ensemble method might not be applicable. Here we borrow the recent advance in uncertainty reasoning techniques – classifier fusion with contextual reliability evaluation (CRE) [11]. It enables characterising refined reliability of a classifier on individual classes and thus allow more effective integration of classifiers.

2.2.1. Classifier fusion with contextual reliability evaluation

CRE is built on top of a formal mathematical theory of evidence, Dempster-Shafer Theory (DST), which propagates uncertainty values and consequently provides an indication of the certainty of inferences. In the following, we will briefly introduce the basics of DST, its application to classification, and then move on to CRE.

2.2.2. Basics of DST

Let $Ω = {h_{1}, \dots, h_{n}}$ be a finite set of mutually exclusive and exhaustive hypotheses, called the frame of discernment. A basic belief assignment (BBA) function assigns belief across the frame; that is, $m : 2^{Ω} \to [0, 1]$ , representing the belief committed to a subset A of Ω, given a certain piece of evidence. $2^{Ω}$ is the power set of Ω; that is, $2^{Ω} = {\emptyset, h_{1} \cup h_{2}, \dots, h_{1} \cup h_{n}, \dots, h_{1} \cup h_{2} \cup \dots h_{n}}$ . The BBA $m (\cdot)$ satisfies the following conditions: $m (\emptyset) = 0$ and $\sum_{A \subseteq Ω} m (A) = 1$ .

Associated with a BBA are belief and plausible functions that define the lower and upper bounds of imprecise probability of a BBA: $bel (A) = \sum_{B \subseteq A} m (B)$ and $pl (A) = \sum_{A \cap B \neq \emptyset} m (B)$ . In another word, for any $A \subseteq Ω$ , $[bel (A), pl (A)]$ is interpreted as the imprecise interval of the unknown probability $P (A)$ , where $bel (A)$ measures that the strength of evidence all supporting A and $1 - pl (A)$ measures that the strength of evidence contradicting A. When there exist multiple independent sources expressing their beliefs through BBA, a common way to combine these BBAs is the following Dempster’s rule in Equation (1). Its principle is to normalise the belief by redistributing the total conflicting belief mass. That is, given two BBAs $m_{1}$ and $m_{2}$ over $2^{Ω}$ , the combination on a nonempty common subset A is defined as $\begin{matrix} (1) & m_{1} \oplus m_{2} (A) = \frac{\sum_{B \cap C = A} m_{1} (B) m_{2} (C)}{1 - \sum_{B \cap C \neq \emptyset} m_{1} (B) m_{2} (C)} . \end{matrix}$

2.2.3. DST application to classification

In a classification problem, each hypothesis in Ω refers to a class of interest. For an input example X, a BBA $m (A)$ characterises a classifier’s belief on assigning the example X to a set of classes. If A is a singular class, it represents the posterior probability; e.g, $m ({y_{i}}) = p (y_{i} | X)$ is the posterior probability of X being classified to the class $y_{i}$ . When A corresponds to a set of classes, $m (A)$ can be used to characterise the imprecision degree among these classes in classification of an input example; e.g., $m ({y_{i}, y_{j}})$ presents the probability of X belonging to either $y_{i}$ or $y_{j}$ , but there is no preference of one or the other. In the extreme case, when $A = Ω$ , $m (Ω)$ denotes the total ignorance degree and it plays a neutral role in the fusion process of multiple sources.

The combination of multiple classifiers’ results can be done through the above Dempster’s rule, which assumes that all the sources are completely reliable. However, each classifier is subject to a certain degree of reliability in that rarely it can produce 100% accuracies of inference on all the examples. To capture the reliability of each classifier, Shafer’s discounting method is often applied. Shafer has defined an evidential operation for discounting the partial mass of belief in a BBA to the total ignorance according to a discounting factor α ( $0 ⩽ α ⩽ 1$ ) [22]. $\begin{matrix} (2) & m^{α} (A) = \{\begin{matrix} α m (A), & A \subset Ω, A \neq Ω, \\ 1 - α + α m (Ω), & A = Ω . \end{matrix} \end{matrix}$ In the above discounting operation, the reliability of a classifier is often described by a single value, and the mass values on different hypotheses are equally discounted. This can be problematic to the SLearn problem, as each dataset can contain training data that are labelled with a subset of activity classes, suggesting that some activities can be more reliably inferred than the others. Contextual reliability evaluation [11] is employed to capture such difference by characterising the reliability of classifiers associated with different contexts (i.e., classes).

2.2.4. Contextual reliability evaluation

CRE focuses on estimating the reliability of each classifier’s inference $m (A)$ and the discounting factor α in Equation (2). The reliability of inference takes into account of the confidence of inferring the current example and the performance of a classifier inferring similar examples. The discounting factor measures the compatibility between the classifiers and helps to find the most plausible class. In the following, we will describe how to estimate the above two parameters.

2.2.5. CRE – Reliability

Given an input example X in a dataset D ( $X \in χ$ ), let $\tilde{f} (X) = y_{i}$ ( $y_{i} \in Y$ ) represent classifying X to a class label $y_{i}$ and $f (X) = y_{i}$ represent the true label X to be $y_{i}$ . A BBA function on the inference result $y_{i}$ is estimated as follows. $\begin{array}{l} m ({y_{i}}) & = r_{ii} * P (\tilde{f} (X) = y_{i} | X) \\ = P (f (X) = y_{i} | \tilde{f} (X) = y_{i}) * P (\tilde{f} (X) = y_{i} | X) \\ (3) & = \frac{P (\tilde{f} (X) = y_{i} | f (X) = y_{i})}{\sum_{l = 1}^{c} P (\tilde{f} (X) = y_{i} | f (X) = y_{l})} * P (\tilde{f} (X) = y_{i} | X) . \end{array}$ The BBA function represents the belief of a classifier on inferring a class label $y_{i}$ . It takes into account both the posterior probability of inferring an input example $P (\tilde{f} (X) = y_{i} | X)$ and the reliability profile $r_{ii}$ of a classifier on each class $y_{i}$ . We cannot directly estimate $r_{ii}$ on X as we do not know the true label of X. A workaround solution is to estimate on the examples similar to X in the training set, whose reliability can serve as a proxy to the reliability profile on X, assuming that a classifier achieves similar accuracies on similar inputs.

To do so, we find the top close neighbours $NG (X)$ to X, and calculate the conditional probabilities $P (\tilde{f} (X^{k}) = y_{i} | f (X^{k}) = y_{l})$ on each neighbour $X^{k}$ ; that is, how often a neighbour $X^{k}$ is classified as $y_{i}$ given that its true label is $y_{l}$ . This conditional probability will then be penalised by the distance between the neighbour $X^{k}$ and X. The intuition is that the closer a neighbour to X, the higher weights will be placed on its reliability measure. The distance penalisation is defined on a commonly applied exponential function with the relative distance of a neighbour $X^{k}$ to X with respect to the minimum distance to the nearest neighbours. The above process is summarised in the following equations. $\begin{array}{l} P (\tilde{f} (X) = y_{i} | f (X) = y_{l}) \\ = \sum_{X^{k} \in NG (X)} P (\tilde{f} (X^{k}) = y_{i} | f (X^{k}) = y_{i}) * δ^{k}, \\ δ^{k} = e^{- λ * d^{k}}, \\ d^{k} = \frac{d (X, X^{k})}{ε + {min}_{X^{k^{'}} \in NG (X)} d (X, X^{k^{'}})}, \end{array}$ where ε is a small constant to compensate, when the minimum distance is 0.

In the end, the reliability profile $r_{ii}$ is a weighted sum of the conditional probabilities on the close neighbours of X. The BBA of a classifier $m (\cdot)$ leverages the reliability of a classifier on different classes and the posterior probability of inferring an example.

2.2.6. CRE – Discounting factor

The second part of CRE is to estimate the discounting factor for each classifier. Dempster’s combination rule in Equation (1) can produce unreasonable results when the sources are highly conflicting [23]. To address this problem, CRE proposes a compatibility measure based on the plausibility functions of each classifier. The criteria of compatibility is to assess whether two classifiers have any common inference. Given two BBAs $m_{i}$ and $m_{j}$ over the same frame of discernment, if there exists a common set of potential classes, then their incompatibility degree is 0; otherwise, it is calculated on the joint product of their maximum plausibility functions. The potential classes $Φ_{i}$ on each classifier are sets of classes that have achieved high plausible functions. They are characterised as the subset of classes if their plausible function is relatively close to the maximum plausible function on the whole frame of discernment. The estimation process is given as follows. $\begin{array}{l} (4) & α_{i} = \frac{{\hat{α}}_{i}}{{max}_{j} {\hat{α}}_{j}}, \\ (5) & {\hat{α}}_{i} = \sum_{j, j \neq i} (1 - κ (i, j)), \\ (6) & Φ_{i} = {A | \frac{{pl}_{i} (A)}{{max}_{B \subseteq Ω} {pl}_{i} (B)} > τ}, \\ (7) & κ (i, j) = \{\begin{matrix} \sqrt{{max}_{A \in 2^{Ω}} {pl}_{i} (A) {max}_{B \subseteq Ω} {pl}_{j} (B)}, & if Φ_{i} \cap Φ_{j} = \emptyset, \\ 0, & otherwise . \end{matrix} \end{array}$

The compatibility-based discounting factor helps to discount the highly conflicting situations between classifiers, which leads to a more effective application of Dempster’s rule. The derived BBA $m_{i} (A)$ and discounting factor $α_{i}$ will be applied to Shafer’s discounting function in Equation (2), whose results will be combined in Dempster’s combination function in Equation (1). In the end, we will infer the class that achieves the highest plausible value; that is, $y \leftarrow \arg \max_{A \subseteq Ω, | A | = 1} pl (A)$ . Algorithm CC (Algorithm 2) describes the classifier fusion process.

Algorithm 2

CC: Combine classifiers

Algorithm 3

SD: Share training data

2.3. Sharing data

A classic approach of dealing with a small amount of training data is leveraging unlabelled data, which is similar to the active learning approach, mentioned in Section 5.2. That is, for each dataset, we train a classifier on its labelled data and then use it to iteratively infer the labels on its unlabelled examples for T rounds or until the algorithm converges. For each iteration, we select the top k most confident examples to enlarge the labelled data pool and iteratively update the classifier. However, given our assumption that the labelled data might be too little and have not covered the whole set of activities of interest, this basic approach can only assign the labels that have been observed in the training data. We will need to leverage labels from the other datasets. This gives rise to Algorithm SD (Algorithm 3). With feature remapping, for each dataset, we will not only train a classifier on its own train data but also on the transferred training data from the other datasets.

2.4. A hybrid of sharing data and classifiers

The advantage of the SD algorithm is to deal with the cold start problem when there exists very few training examples in each dataset. It helps to accumulate training examples, but it ignores the individual users’ activity routines and patterns. This can lead to negative transfer; that is, learning across datasets might result in decreased accuracies. To prevent the situation, we proposed the SDC algorithm (Algorithm 4) to share data and classifiers in an informed way. The principle is that if the performance of a dataset’s classifier is not improved any more with sharing data, then we should stop using the other datasets’ data for training any more and switch to sharing classifiers. There are different ways to measure a classifier’s performance; for example, the overall accuracies measure how accurately it infers a correct class label. Here we opt for the class accuracies as sharing data aims to improve the diversity of classes on each dataset and the class accuracies will not biased towards the classes observed in each dataset and will provide more accurate view on the overall performance of the classifier.

Algorithm 4

SDC: An informed combination of SD and SC

3. Experiment setup and evaluation methodology

The objectives of the evaluation is to assess the accuracy of SLearn recognising activities by shared learning on datasets that are only partially annotated with activities of interest. More specifically, we aim to answer the following questions:

Does SLearn outperform the classic supervised and semi-supervised learning techniques in HAR?

Which SLearn algorithm is more effective and under what circumstances?

Does CRE outperform DST, in terms of supporting robust classifier fusion in the face of high conflict and uncertainty?

We measure the accuracy in overall accuracy – the ratio of the learnt labels being the same as the true labels, and class accuracy – the averaged ratio of the learnt labels being the same as the true labels in each class $y \in Y$ . The class accuracy is more indicative when the datasets have imbalanced class distribution, which is a common problem with the HAR datasets [6].

3.1. Datasets

To test SLearn, we need to find the datasets that are collected in different environments but have the common set of activities to allow for shared learning. Therefore, we locate the three HAR datasets collected by the University of Amsterdam [26], which are widely used in the literature. These datasets were collected in real-world houses with different spatial layouts, a different number of sensors, and the degree of inherent noise. We believe that experimenting on these datasets gives us a comprehensive view of the effectiveness of the proposed technique.

These three datasets recorded the same set of 7 activities, including leaving the house, preparing breakfast or dinner, and sleeping. The activity distribution is presented in Fig. 3, which shows that the occurrence of the activities is imbalanced and each house has a dominant activity. These three houses are deployed with only binary sensors and a more detailed description of the datasets can be found in [8]. The House A dataset consists of 14 state-change sensors attached to household objects like doors, cupboards, and toilet flushes, while the other two datasets contain more than 20 sensors, including reed switches to measure whether doors and cupboards are open or closed; pressure mats to measure sitting on a couch or lying in bed; mercury contacts to detect the movement of objects (e.g., drawers); passive infrared to detect motion in a specific area; float sensors to measure the flush of toilet. The sensor metadata are also provided along with the dataset, and some of which are presented in Fig. 3. The metadata allows us to map sensor on the Location and Object ontological concepts and facilitate the feature space remapping process.

Fig. 3.

Activity class distributions and sensor metadata of the three datasets used for experimental evaluation.

3.2. Model configuration and evaluation process

As SLearn aims to explore how little the training data is required to build a robust activity model, we are using the shallow learning techniques as base classifiers, including Naive Bayes (NB), Random Forest (RF), Logistic Regression (LR), Neural Networks (NN), and Support Vector Machine (SVM).

3.3. Evaluation process

The key objective of SLearn is to integrate partially annotated, heterogeneous datasets to recognise a better covered set of activities. To do so, we evaluate the algorithms across different ratios of training data, from 0.1% to 80%. 0.1% is the extreme training data percentage for the chosen datasets; that is, the training data for each dataset can only contain 1 or 2 examples. This extreme percentage can be too little to train a classifier properly, but we would like to build a comprehensive performance profile of SLearn and decide what training percentage is appropriate to SLearn.

The hypothesis of SLearn is that if the examples from each dataset covers different activities; e.g., ‘leave home’ from House A and ‘use toilet’ from House B, SLearn will be able to recognise both activities on the test data in House A and B, while a traditional supervised activity recognition algorithm will only be able to recognise ‘leave home’ in House A and ‘use toilet’ in House B. Here we start with the low training ratio and systematically increase the ratio to find the optimal ratio of training data with which SLearn can achieve reasonable accuracies.

4. Results and discussion

This section discusses the evaluation results to address the research questions in Section 3.

4.1. SLearn model configuration

Our first experiment is to understand the performance of SC by evaluating the impact of different base classifiers and distance measures on their accuracy. Figure 4 presents the instance and class accuracy of SC on House A, B and C, when we randomly sample 2% training instances from each dataset to run SC. We have run the experiments on all the different number of training instances and they have the similar pattern as Fig. 4.

The results show that none technique or distance measures significantly outperforms the others, except for that SVM performs the worst among all. This suggests that SC is not sensitive to the choices of these two configurations. In the following, we will report the results on NB and RF for the following reasons. These two algorithms represent two different styles of classifiers: NB is a simpler model and consumes less training data to produce good classification accuracies, while RF has more capacity in building complex models but requires more training data. We want to see how these two algorithms work when the training data is little and how all the algorithms can enhance their performance in this situation.

Fig. 4.

SC accuracy with different base classifiers and distance metrics in CRE when the percentage of training sample is 2%. We report mean and standard deviation of instance and class accuracy. We mark the highest mean accuracy as bold, per distance measure, per classifier, and overall. The distance metrics are Euclidean EU, Cosine CS, Manhattan MA, and Kullback–Leibler divergence KL.

Fig. 5.

Comparison of overall and class accuracies of CRE- and DST-based classifier fusions.

4.2. Comparison of CRE and DST in SC

In our previous work [30], we have evaluated SC Algorithm and compared with the classic ensemble methods such as majority voting, highest confidence and DST to integrate imperfect evidences from distinct sources [13]. All three have achieved similar results and DST has more consistently better performance. Here we compare the averaged overall and class accuracies of Algorithm SC with CRE and DST in Fig. 5. In general, CRE-based Algorithm SC performs better, suggesting that the inclusion of classifier reliability and compatibility does help to fuse classifiers more robustly. With the increase of training data, the difference between these two gets smaller. The reason is that when the training data is sufficient, the classifier will be less uncertain about their inference and will be more likely to use their own classifications, which leads to similar accuracies towards the end spectrum of the training data.

In some cases, DST works better, especially when the training data is very small. For example on House B, when the percentage of training data is 0.1%, DST-based Algorithm CC works best. This can be due to the noise in the data where different activities share the same sensor signature. In this case, if the closest neighbour to the current example is labelled with a different activity, then the estimated reliability in Equation (3) can compromise the posterior probability of the classifier inferring the example. However, when the training data increases, there are more candidates in the training set to be selected as neighbours, and the impact of the noise reduces, leading to better performance of CRE.

4.3. Comparison of SLearn and baseline techniques

Figure 6 compares overall and class accuracies of SDC, SC, SD, (B)aseline, and (A)ctive learning algorithms. SLearn Algorithms improve both overall and class accuracies when the ratio of training data is small and perform comparably with the baseline approaches afterwards.

Fig. 6.

Comparison of overall and class accuracies of SDC, SC, SD, (B)aseline, and (A)ctive learning algorithms.

We can see that even when the ratio of training data is only 0.1%, the overall accuracies on the baseline approaches can be as high as 50%, especially on House B and C. After looking into the inference results, we find that on these two datasets, if the training data only contains the high frequency activities, then both the classifiers NB and RF will only predict these two activities, leading to the final overall accuracies to be the actual class distribution. This also explains why on House B, Algorithms SDC and SC do not achieve higher overall accuracies than Baseline approaches when the percentage of training data is less than 0.4%. When the training data are from the dominated activities, the classifier on individual datasets will work better, without losing the specifics of sensor signature during the sensor feature remapping or compromising accuracies with the training data from the other datasets.

Fig. 7.

Confusion matrices on House A with SD, SC, and baseline algorithms on naive Bayes and random forest.

To gain a deeper insight, we plot the confusion matrices of SDC, SC, and Baseline algorithms on Naive Bayes and Random Forest in Fig. 7, when one example from House A, B, and C is selected: ‘toilet’, ‘leave house’, and ‘sleep’ respectively. These activities are dominant in each dataset, as shown in Fig. 3. With the baseline classifiers, especially RF, the classifier is completely biased towards the activity being observed. For NB, if a test example is not similar to the observed instance, then it assigns the evenly distributed probabilities to all the activities, which result in the first class always being inferred. On the contrary, SD and SC algorithms have exhibited the ability to gather and integrate the examples from each dataset and make more meaningful and better covered predictions. For example, on House A, both SLearn algorithms can recognise the ‘leave home’ and ‘sleep’ activities, which are not in the training data of House A but learnt from House B and C.

We have experimented three uncertainty sampling strategies of active learning including least confidence, margin of confidence, and information entropy, and the uncertainty entropy strategy produces the best accuracy. The performance of active learning with the information entropy uncertainty sampling strategy is presented in Fig. 6, including the times of queries (Q) to acquire labels on uncertain examples from human operators and the overall and class accuracies. Active learning achieves better accuracies, however, it requires to query 10% or 20% of test data to human operators, which makes the actual amount of training data is actually much higher than the specified.

4.4. Discussion

4.4.1. Comparison between SLearn algorithms

When the training data is small, sharing data works better than sharing classifiers. As presented in Fig. 6, Algorithm SDC and SD achieves higher overall and class accuracies than Algorithms SC when the training percentage is less than 1%. It suggests that sharing data expands the training data for each dataset and improves the activity coverage, which helps to boost the performance. Sharing data can enhance activity coverage, but not necessarily improve identifying activities for individual users. Both overall and class accuracies of Algorithm SD do not change much with the increase of training data, especially on House B and C. This observation justifies Algorithm SDC that leverages sharing data when the training data is insufficient and switches to sharing classifiers only when the current classifier is uncertain about their inference.

Algorithm SC needs more training data than Algorithm SDC and SD to perform well, as there is no significant improvement when the training ratio is very small. First of all, Algorithm SC still trains its own classifier on training data from its own dataset; that is, the training data size is not enlarged like Algorithm SDC and SD. But when the training ratio increases, SC has better improvement. Secondly, the small training data significantly undermines the estimation of the uncertainty threshold. This becomes a more severe problem on RF than NB.

4.4.2. Challenges in activity recognition datasets

In the experiments, we have encountered two challenges in these activity recognition datasets: (1) imbalanced class distribution and (2) the variety of patterns in activities. The imbalanced class distribution often leads the high overall accuracies as the training data is more likely to contain the examples from the dominated, frequent activities. However, it does not help improve class accuracies. In the future, various sampling techniques can be applied to either under-sampling frequent activities or over-sampling infrequent activities. Also, recent data augmentation techniques [4,14,24] can be applied to either simulate data samples and augment data with variations.

Each activity can have a variety of patterns, and some activities can have similar or the same patterns. Often we need a large amount of training data to learn the difference so as to be able to distinguish them. This will be a particular challenge to SLearn. We try to address this challenge by introducing contextual reliability evaluation based classifier fusion. However, when the training data is too small, the algorithm fails to locate enough neighbours to draw a reliable conclusion. When the neighbours are labelled with a different activity, then the reliability scores are compromised.

4.4.3. Comparison with active learning algorithms

Figure 6 has shown that active learning outperforms SLearn algorithms. In addition to the high query ratio, another reason for the better performance is that active learning assumes the always availability of true labels for each user’s own sensor data. On the one hand, this allows annotating sensor data for individual users and thus improves their own activity recognition model. On the other hand, it has better coverage of the activity space than our proposed methods in that the labels that we share in SD and SC are constrained by their availability in the other datasets’ training data. For example, if all the datasets only contain ‘having a meal’ and ‘sleeping’ activities altogether, then our techniques will not be able to detect any activities other than these two, but the active learning technique will still be able to learn the others such as ‘taking shower” or ‘having drink’ because of the always availability assumption.

4.4.4. Knowledge engineering effort

The smart home ontologies that we are using are light-weight and generic to various smart home environments without any modifications, which has been demonstrated in our previous work [32,33]. To perform feature space remapping, we take the sensor deployment file to map sensors to their corresponding location and object concepts.

However, we understand that not all of the environments have well-recorded sensor deployment files, which will involve some manual labelling on sensors. Also if a sensor is removed or moved, or a new sensor is introduced, we will need to remap sensors and this effort is unavoidable in our current design. This limits the application of our knowledge-driven approach.

5. Related work

Activity recognition has been an active research topic in the last decade, and a large number of knowledge- and data-driven techniques have been proposed [31]. Among them, different approaches have been designed to address the scarcity challenge of activity annotation, including unsupervised learning, activity learning, and transfer learning. In the following, we will compare and contrast these approaches with SLearn.

5.1. Unsupervised learning

Unsupervised learning automatically partitions and characterises sensor data into patterns that can be mapped to different activities without the need of annotated training data. Pattern mining and clustering are the two mostly used techniques that support unsupervised activity recognition. Gu et al. have applied emerging patterns to mine the sequential patterns for interleaved and concurrent activities [7]. Rashidi et al. propose a method to discover the activity patterns and then manually group them into activity definitions [20]. Based on the patterns, they create a boosted version of a Hidden Markov Model (HMM) to represent the activities and their variations in order to recognise activities in real time. Similarly, Ye et al. have combined the sequential mining and clustering algorithms to discover representative sensor events for activities. Different from the work in [20], they have applied the generic ontologies to automatically map the discovered sensor sequential patterns to activity labels through a semantic matching process [32]. Yordanova et al. have also applied domain knowledge in rule-based systems to generate probabilistic models for activity recognition [9,35].

Taking a different route, researchers also have applied web mining and information retrieval techniques to extract the common sense knowledge between activities and objects via mining online documents; that is, what objects are used to perform a daily activity and how significant each object is contributed to identifying this activity [17,29,34,36]. During the reasoning process, the mined objects are mapped to sensor events and an appropriate activity will be recognised.

SLearn is not a classic activity recognition problem where for each dataset a model is trained with labelled instances and used to recognise unlabelled instances. In SLearn, the training data is assumed to be incomplete in that it might not cover all the activities of interest. SLearn is not a unsupervised learning technique as it still relies on labelled training data, but the labels can come from different datasets.

5.2. Active learning

Active learning, so called ‘query learning’, is a subfield of machine learning, which is motivated by the scenario when there is a large amount of unlabelled data but a limited and insufficient amount of labelled data. As the labelling process is tedious, time-consuming and expensive in real-world applications, active learning methods are employed to alleviate the labelling effort by selecting the most informative instances to be annotated [21].

Alemdar et al. apply active learning strategies to select the most uncertain instances to be annotated; that is, the instances sit at the boundaries of different activity classes [1]. The annotated instances are then used to iteratively update a HMM to infer daily activities. The active learning strategies have improved recognition accuracies, compared to random selection. Cheng et al. apply a density-weighted method that combines both uncertainty and density measures into an objective function to select the most representative instances for user annotation [3].

SLearn is better than an active learning problem [21] in that the latter still needs to query a user or a human operator, but SLearn queries the labels from the other datasets. However, we could use the uncertainty sampling strategies in active learning to determine when an algorithm should leverage training data or classifiers from other datasets.

5.3. Transfer learning

Want et al. [27] propose a stratified transfer learning to improve accuracies of cross-domain activity recognition. Here, cross-domains mean different positions where the accelerometers are placed. The key idea is to exploit the intra-affinity of classes to perform intra-class knowledge transfer.

Maekawa et al. [12] propose an unsupervised approach to recognise physical activities from accelerometer data. They utilise information about users’ characteristics such as height and gender to compute the similarity between users, and find and adapt the models for the new users from the similar users. van Kasteren et al. [25] propose a manual mapping between sensors in different households and learn the parameters of a target model using the EM algorithm to transit probabilities of HMM models from source to target. Similarly, Rashidi et al. [19] learn sensor mappings based on their locations and roles in activity models. The role is characterised in mutual information, measuring the mutual dependence between an activity and a sensor and suggests the relevance of using the sensor in predicting the corresponding activity. Feuz et al. [5] propose a data-driven approach to automatically map sensors based on their meta-features, which are mainly about when a sensor reports, and time intervals between events reported by this sensor and others.

SLearn works on a different assumption from the above works where they assume a complete model (that is, containing all the activities of interest) can be learnt on a source domain, while we assume each domain can only have a small fraction of data being annotated (that is, the activities having been annotated can be a subset of activities of interest in a domain) and we do not assume any domain necessarily as a source or target domain. However, transfer learning techniques such as feature remapping can be applied to SLearn. Especially, our approach is most similar to the above three, where we focus on sensor mappings to support sharing sensor data across multiple datasets. The difference is that we are using a knowledge-driven approach where sensors are modelled in location and object ontologies whose generality across different households and sensing technologies has been demonstrated in other works [32].

6. Conclusion and future work

This paper proposes a set of SLearn algorithms to address the problem on the scarcity of activity annotations by leveraging annotations across different datasets that can have different sensing technologies, different sensor deployment, and different users, as long as they share the same mission with compatible sensor features and activities of interest.

The evaluation results have consistently demonstrated a significant improvement on recognition accuracies when training data is small. The success of SLearn can scale activity recognition applications by leveraging each user’s sporadic annotations.

The future work is to resolve heterogeneity of activity labels in different datasets. For example, given that two datasets have annotations on ‘preparing breakfast’ and ‘preparing dinner’, can we combine them to recognise ‘making a meal’ on the third dataset? To address this question, we will first define the semantic similarity between different activity labels [32]; that is, specifying an activity with the common location and object ontologies. Then we can look into an instance transferring approach to factor the activity similarity in. For example, we will start with the recent approach that has the capacity of borrowing examples from similar but not the same classes by boosting the loss function with similarity weight for a target class in the objective function [10]. We will also adapt the current approach to accelerometer data-based activity recognition.

References

Alemdar,

T.L.

van Kasteren and

Ersoy, Using active learning to allow activity recognition on a large scale, in: International Joint Conference on Ambient Intelligence, Springer, 2011, pp. 105–114. doi:10.1007/978-3-642-25167-2_12.

Chen and

Xue, A deep learning approach to human activity recognition based on single accelerometer, in: 2015 IEEE International Conference on Systems, Man, and Cybernetics, 2015, pp. 1488–1492. doi:10.1109/SMC.2015.263.

H.-T.

Cheng,

F.-T.

Sun,

Griss,

Davis,

Li and

D.Y.

Nuactiv, Recognizing unseen new activities using semantic attribute-based learning, in: MobiSys 2013, 2013, pp. 361–374.

O.S.

Eyobu and

D.S.

Han, Feature representation and data augmentation for human activity classification based on wearable IMU sensor data using a deep LSTM neural network, Sensors 18(9) (2018), 2892. doi:10.3390/s18092892.

K.D.

Feuz and

D.J.

Cook, Transfer learning across feature-rich heterogeneous feature spaces via feature-space remapping (FSR), ACM Trans. Intell. Syst. Technol. 6(1) (2015), 3:1–3:27. doi:10.1145/2629528.

K.D.

Feuz and

D.J.

Cook, Collegial activity learning between heterogeneous sensors, Knowledge and Information Systems 53(2) (2017), 337–364. doi:10.1007/s10115-017-1043-3.

Gu,

Wu,

Tao,

H.K.

Pung and

Lu, epSICAR: An emerging patterns based approach to sequential, interleaved and concurrent activity recognition, in: PerCom 2009, 2009, pp. 1–9.

T.L.M.

Kasteren,

Englebienne and

B.J.A.

Kröse, Human activity recognition from wireless sensor network data: Benchmark and software, in: Activity Recognition in Pervasive Intelligent Environments, Atlantis Ambient and Pervasive Intelligence, Vol. 4, Atlantis Press, 2011, pp. 165–186. doi:10.2991/978-94-91216-05-3_8.

Kruger,

Nyolt,

Yordanova,

Hein and

Kirste, Computational state space models for activity and intention recognition. A feasibility study, PLOS ONE 9 (2014), 1–24.

10.

J.J.

Lim,

Salakhutdinov and

Torralba, Transfer learning by borrowing examples for multiclass object detection, in: NIPS ’11, Curran Associates Inc., USA, 2011, pp. 118–126.

11.

Liu,

Pan,

Dezert,

J.W.

Han and

He, Classifier fusion with contextual reliability evaluation, IEEE Transactions on Cybernetics 48(5) (2018), 1605–1618. doi:10.1109/TCYB.2017.2710205.

12.

Maekawa and

Watanabe, Unsupervised activity recognition with user’s physical characteristics data, in: 2011 15th Annual International Symposium on Wearable Computers, 2011, pp. 89–96. doi:10.1109/ISWC.2011.24.

13.

Mckeever,

Ye,

Coyle,

Bleakley and

Dobson, Activity recognition using temporal evidence theory, J. Ambient Intell. Smart Environ. 2(3) (2010), 253–269. doi:10.3233/AIS-2010-0071.

14.

Mikojczyk and

Grochowski, Data augmentation for improving deep learning in image classification problem, in: 2018 International Interdisciplinary PhD Workshop (IIPhDW), 2018, pp. 117–122. doi:10.1109/IIPHDW.2018.8388338.

15.

G.A.

Miller, Wordnet: A lexical database for English, Communications of the ACM 38(11) (1995), 39–41. doi:10.1145/219717.219748.

16.

Münzner,

Schmidt,

Reiss,

Hanselmann,

Stiefelhagen and

Dürichen, Cnn-based sensor fusion techniques for multimodal human activity recognition, in: Proceedings of the 2017 ACM International Symposium on Wearable Computers, ISWC ’17, ACM, New York, NY, USA, 2017, pp. 158–165. doi:10.1145/3123021.3123046.

17.

Palmes,

H.K.

Pung,

Gu,

Xue and

Chen, Object relevance weight pattern mining for activity recognition and segmentation, Pervasive and Mobile Computing 6(1) (2010), 43–57. doi:10.1016/j.pmcj.2009.10.004.

18.

S.J.

Pan and

Yang, A survey on transfer learning, IEEE Transactions on Knowledge and Data Engineering 22(10) (2010), 1345–1359. doi:10.1109/TKDE.2009.191.

19.

Rashidi and

D.J.

Cook, Multi home transfer learning for resident activity discovery and recognition, in: KDD Knowledge Discovery from Sensor Data, 2010, pp. 56–63.

20.

Rashidi,

D.J.

Cook,

L.B.

Holder and

Schmitter-Edgecombe, Discovering activities to recognize and track in a smart environment, IEEE Transaction on KDE 23(4) (2011), 527–539.

21.

Settles, Active learning literature survey, Computer Sciences Technical Report 1648, University of Wisconsin–Madison, 2009.

22.

Shafer, A Mathematical Theory of Evidence, Princeton University Press, 1976.

23.

Tchamova and

Dezert, On the behavior of dempster’s rule of combination and the foundations of Dempster–Shafer theory, in: 2012 6th IEEE International Conference Intelligent Systems, 2012, pp. 108–113. doi:10.1109/IS.2012.6335122.

24.

T.T.

Um,

F.M.J.

Pfister,

Pichler,

Endo,

Lang,

Hirche,

Fietzek and

Kulić, Data augmentation of wearable sensor data for Parkinson’s disease monitoring using convolutional neural networks, in: ICMI ’17, ACM, 2017, pp. 216–220. doi:10.1145/3136755.3136817.

25.

T.L.M.

van Kasteren,

Englebienne and

B.J.A.

Kröse, Transferring knowledge of activity recognition across sensor networks, in: Pervasive 2010,

Floréen,

Krüger and

Spasojevic, eds, 2010, pp. 283–300.

26.

T.L.M.

van Kasteren,

Englebienne and

B.J.A.

Kröse, Human activity recognition from wireless sensor network data: Benchmark and software, in: Activity Recognition in Pervasive Intelligent Environments, Atlantis Press, Paris, 2011, pp. 165–186.

27.

Wang,

Chen,

Hu,

Peng and

P.S.

Yu, Stratified transfer learning for cross-domain activity recognition, in: 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom), 2018.

28.

Wu and

Palmer, Verbs semantics and lexical selection, in: ACL ’94, Association for Computational Linguistics, Stroudsburg, PA, USA, 1994, pp. 133–138.

29.

Wyatt,

Philipose and

Choudhury, Unsupervised activity recognition using automatically mined common sense, in: AAAI ’05, AAAI Press, 2005, pp. 21–27.

30.

Ye, Slearn: Shared learning human activity labels across multiple datasets, in: 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom), 2018, pp. 1–10.

31.

Ye,

Dobson and

McKeever, Situation identification techniques in pervasive computing: A review, Pervasive and mobile computing 8 (2012), 36–66. doi:10.1016/j.pmcj.2011.01.004.

32.

Ye,

Stevenson and

Dobson, Usmart: An unsupervised semantic mining activity recognition technique, ACM Trans. Interact. Intell. Syst. 4(4) (2014), 16:1–16:27.

33.

Ye,

Stevenson and

Dobson, Kcar: A knowledge-driven approach for concurrent activity recognition, Pervasive and Mobile Computing 19 (2015), 47–70. doi:10.1016/j.pmcj.2014.02.003.

34.

Yordanova, From textual instructions to sensor-based recognition of user behaviour, in: IUI ’16 Companion, ACM, New York, NY, USA, 2016, pp. 67–73. doi:10.1145/2876456.2879488.

35.

Yordanova and

Kirste, A process for systematic development of symbolic models for activity recognition, ACM Trans. Interact. Intell. Syst. 5(4) (2015), 20:1–20:35.

36.

V.W.

Zheng,

D.H.

Hu and

Yang, Cross-domain activity recognition, in: Proceedings of the 11th International Conference on Ubiquitous Computing (Ubicomp ’09), ACM, 2009, pp. 61–70. doi:10.1145/1620545.1620554.