Computational methods for predicting human behaviour in smart environments

1.1. Background

This systematic review of computational methods of human behaviour prediction examines key algorithms and datasets with the purpose of determining what the most relevant advancements are for computationally predicting human behavioural outcomes. The review is bounded from 1999 to 2022, starting with the highly cited Pentland and Liu paper [49] which began the new generation of behaviour prediction work in computer science, and includes research up to the present day, at the time of writing. This review addresses a gap in the literature of an aggregate of the key work in this area, presented as an accessible summary of the important sub-fields and computational methods, and provides a starting point for researchers new to this field. The authors used this literature review as the foundation for behaviour prediction research at the University of Manchester. It was first conducted in 2018 and then updated in 2022 to share with the wider research community. As such the data tables include both the original and updated review papers and the aggregate data from both in the analysis.

1.2. Key concepts in behaviour prediction

The behaviour prediction field builds upon work in object recognition, action recognition, and behaviour modelling, also known as Human Activity Recognition (HAR). Action or object recognition uses discrete classifiers to determine an object or event such as dog, cat, human or waving, walking, running – for example. This is usually achieved through computer vision analysis of images or video combined with a classification model. These datasets are built up by using human classifiers to label video or images so they can be correctly validated to determine the model accuracy. Once we have these discrete attributes and an ability to classify them it becomes possible to model the behaviours of humans, perhaps using a knowledge graph or other suitable data structure. For example, we can model the behaviour of a human by connecting its action attributes such as talk, eat, walk, run, sit, and sleep. We now have a human model, but how do we know what it will do next? A prediction algorithm takes the features of human behaviour and calculates – through various different methods as detailed later in this paper – and determines which behaviour, or state, will happen next. These techniques are known as Human Activity Prediction (HAP), and are often a probabilistic state change value determined by previously observed data of the human’s behaviours and how one state can lead to another. For example ‘run’ will rarely precede ‘sleep’, but ‘eat’ may often occur before ‘sleep’ and so it will have a higher likelihood of being the next state after eating.

Much of the work by the research community in recent years has focussed on solving the foundational problem of activity recognition [14,18,24,25,50], which is instrumental in providing the labelled data for prediction models. All of the predictive methods found in this review build upon the work done by the research community in activity recognition, and it is important to understand the dependent relationship between HAR and HAP, as shown in the diagram in Figure 1.

Action and object recognition are the building blocks or precursor that we use to generate the base data to understand what a human is doing. These algorithms are out of scope of this review, however, all we need to understand is that action recognition algorithms generate the labelled sequence of events used for modelling. Connecting these singular classified events together creates a model of actions or events we understand as behaviours. These models can be sequential time series tables, or they could be graph-like state change models. Prediction algorithms then use these models to calculate which is the most likely next event or activity.

OPEN IN VIEWER

Fig. 1.

The process of next event prediction.

The objective of this review was to create a summary of the key information in this field needed for further computer science research. To achieve this we set the three research questions below, which will provide the foundation level of knowledge needed to conduct novel investigations in human behaviour prediction. These questions will determine these pieces of information: datasets, modelling, and prediction algorithms.

RQ1 What are the public domain datasets used for behaviour prediction in smart environments?

Answered in Section 4.1

1.4. Review methodology

We used the PRISMA checklist for selection and review of papers, and modified the results and discussion sections to exclude areas that are focussed on clinical research rather than computer science. You will see this reflected in the review sections later in this paper. For this review we wanted to answer the knowledge discovery research questions in Section 1.3. A granular examination of each dataset and algorithm is out of scope for this review but it would be a good research opportunity for future work. As such this review presents a high-level overview of the key information, with relevant comparisons and elaboration where necessary.

2.1. Protocol

The search and selection have all followed the PRISMA guidelines,1

The eligibility criteria used for this systematic review are as follows.

These criteria are derived from Boland et al. [10] and were agreed with the co-reviewers before proceeding to the search and selection stage.

2.3. Information sources

The following databases have been searched: Google Scholar, ACM digital library, IEEE Xplore, ScienceDirect, SpringerLink, and Wiley Online Library. This list of databases was compiled from the sources found in the scoping search in Google Scholar, which aggregates papers from different sources. This included the most recognised platform for computer science research (ACM), along with well known publishing sources (IEEE, Springer, Wiley, and Elsevier (ScienceDirect)).

For the scope of this systematic review the databases listed above are comprehensive and cover a large amount of research material to both reduce bias, and be reassured that a reasonable effort has been made to cover all the significant material.

2.4. Search

The search and selection process followed those detailed in the PRISMA guidelines. The keyword search used ‘human behavior prediction’ as its base search terms – from the title of the 1999 Pentland and Liu paper [49], along with combinations of synonyms and spelling variations (U.S. and British) as seen in Table 1. Boolean operators were also used to improve the accuracy of search results.

The search string was constructed using the synonyms in each column of Table 1 and combined into one boolean string.

Each database was searched for all papers that matched the search criteria, before being filtered using the inclusion criteria in Section 2.2 and the quality assessment below.

The main literature search and de-duplication was conducted using a Python script that either calls the database API directly, uses the Crossref API, or uses web scraping to acquire the search data. The script requests the first 1000 entries for the search string from each database (or fewer if less than 1000 are available). Resulting in a maximum of 6000 papers for this review.

The lists of papers were then de-duplicated, and at the same time filtered for some of the inclusion criteria: published date, and keywords in the title. This part was quite aggressive in only including results with the string predict or forecast in the title, and as such reduced the number of results from 3360 to 134. The results for each stage of the process for the main database search and selection are shown in Table 2.

The Python search code is available online at Github,4

To check this process was not excluding relevant papers, 33 papers (approximately 1% of the initial search results) were randomly selected from the list of 3360. The papers were selected using a random number generator in a Python script that only selected papers that had previously been excluded. The random number corresponded to the spreadsheet row number in the list of papers. The results of the test to validate the filtering process show that none of the 33 random papers selected passed the eligibility criteria – therefore validating the title keyword and publication date filter. The list of random papers, Python script, and inclusion criteria test are available online.5

Finally each paper was assessed and awarded points based on the following criteria: 1 point if it meets the criteria, 0.5 points if it partly meets the criteria, or 0 (zero) if it does not meet the criteria. Papers with a score of less than 4 (four), a QA of less than 80%, were rejected.

These criteria were derived from Kofod-petersen [33], and agreed with the co-reviewers verifying the protocol, as appropriate for filtering out low quality papers.

To minimise subjective bias in the review process, the review search and selection (including inclusion criteria and quality assessment) was conducted by three PhD researchers from the University of Manchester’s Interaction Analysis and Modelling Lab.6

The data to answer the systematic review objectives was extracted from each of the final papers into a spreadsheet. High level attributes of each study plus prediction accuracy were sufficient for the data summary performed here to produce a list of methods and algorithms, and their respective accuracies.

2.7. Data items

The data collected from each source: Input data type, input dataset, data structure used to represent behaviour, processing method/algorithm used, output model or data, prediction accuracy (%), results analysis used, use cases stated, keywords.

3.1. Data extraction

The key data extracted from the literature includes the smart home datasets used; prediction algorithms; data structures used for modelling behaviour; and input data types e.g. sensors, video, location data; and keywords and use cases from each of the papers e.g. Ambient Assistive Living (AAL). The key extracted data can be found in Tables 3, 4, 5, and 6.

Full research data extracted from the final papers (listed in Table 7 is available online for re-use, validation, and further research.7

Many of the studies use freely available datasets from research universities such as MIT or Washington State. These are Mavhome,8

The datasets contain different input data types from smart environments, see Fig. 2, including sensors placed around a home, power consumption of different appliances, device operations, location data, video, and accelerometers. A comparison of the dataset to highest recorded prediction accuracies in Table 8 shows that MavHome, CASAS, LIARA, and Placelab have generated comparable high accuracies from their data.

OPEN IN VIEWER

Fig. 2.

Input data types.

OPEN IN VIEWER

Table 8

Dataset/algorithm prediction accuracies

Choosing a data structure to model behaviour is an important precursor to behaviour prediction. We can see in Table 9 that the literature shows behavioural activities are modelled in five different ways: Graphs, Markov graphs, Time-series tables, Tree, and Vector. Fundamental to modelling this type of data is representing the sequential or temporal nature of the data in the data structure.

Vectors are the simplest of all these data structures. We can imagine a sequence of events represented as sensor values ( 27 , 14 , 1 , 903 , 43 , 17 ) and a vector is simply a 1-dimensional representation of some sequence of behavioural events. To enrich this data we may want to create a 2-dimensional representation of the data. This is where time-series event tables are used to create a matrix of the events, such as annotating those sensor values with ‘occupant’, ‘sensor source’, and ‘location’, for example.

To model more complex relationships between events, activities, and data points within a smart environment we must look at data structures that can support a higher number of dimensions, and be extensible to advanced algorithmic processing. Graph data structures are an ideal candidate and lend themselves well to behaviour modelling where vertices represent activities or events, and edges represent the connections or transitions between them. A graph can show causality by following the chain of vertices, and is incredibly versatile in enabling calculations to be performed on the edges or on the vertices using many different computational methods. Graphs also have notable subsets which include Markov variants that can reduce down the number of states by grouping them, so only the current state or n-th number of states is considered in a transition rather than a longer sequence of events. Trees are another type of graph, acyclic in nature, that represent the behavioural data in the opposite way to Markov graphs, in that they are purely causal in showing how one event leads to another. Graph data structures and combinations of different types of graphs have the most potential to computationally represent highly complex behaviours.

Prediction algorithms or methods fall into three categories: Machine Learning (ML), Probabilistic Graphical Models (PGMs), and Statistical and Trend Analysis [62]. Machine learning algorithm variants include Reinforcement Learning (RL) [4,28], Neural Networks (NN) [2,38,61] [45], Recurrent Neural Networks (RNN) [5], Long Short Term Memory (LSTM) [5,29], and Support Vector Machines (SVM) [29]. Probabilistic Graphical Models include SPEED algorithm variants (sequence prediction via enhanced episode discovery) [3,22,43] and CRAFFT algorithm variants (current activity and features to predict next features/Bayesian network) [48]. Lastly, Statistical and Trend Analysis which includes Facebook’s Prophet, ARIMA, and SARIMA [55,57]. A full list of algorithms can be found in Table 3, and a simple relationship between data structures and algorithms can be found below in Table 10.

The prediction accuracies range from 43.9 to 100, as seen in Table 3. However, the studies were predicting different behaviour types so future research would need to differentiate based on their use case. Some studies focussed on complex behaviours such as Activities of Daily Living (ADL), whereas others simply predicted the next location a person would occupy. The use cases and keywords in Table 6 demonstrate this further.

Table 8 shows a comparison of the different algorithm’s prediction accuracies (%) for each of the datasets used, where the highest stated prediction accuracy for each dataset/algorithm combination is used for each table entry.

Accuracy expressed as a percentage of correct predictions (%) features most commonly in the results of this review. This is standard way of assessing the efficacy of classification tasks. However, prediction tasks with a continuous output tend to use Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) which appears in a number of papers, and denotes the error distance from the regression line of best fit, where a low number equates to better accuracy.

3.6. Output type

The majority of the systems output a single value – the prediction. This could be: an action, a probability, a location, a health data value, an appliance or device usage. None of the systems detailed in the studies used more complex data structures, such as matrices, to output multiple data points (multiple time based predictions) at the same time.

3.7. Use cases stated

All of the uses cases stated, see Table 6, can be categorised as smart environments, with distinct sub-categories that focus on Activities of Daily Living (ADL) or Ambient Assistive Living (AAL).

4.1. What are the key datasets?

The majority of current research in this area focusses on the MavHome [19,31,51], MavLab [3,30], CASAS [13] [23,45,47,59], LIARA [11,34], and Placelab [15,32] datasets from MIT and Washington State University. Some novel datasets exist which report excellent results, as shown in Table 8, specifically the Shanghai smart home dataset and Neural Network combination which reported a perfect score [63].

As expected the input data from the datasets are a mix of types: sensors including motion sensors, actuators, and switches; appliances such as kettles and cookers; devices like thermostats; and more complex data input such as accelerometers, location data, and less commonly video.

The relatively low use of video is interesting given the advances in computer vision. The processing of video for behaviour prediction may have been too computationally expensive, something that has changed only very recently. Or there could be cultural and privacy concerns that make researchers believe it is unlikely that a video feed in every room of a home is ever likely to be accepted by the public, and therefore that research is not worth pursuing.

Furthermore, the complete absence of audio as an input method is also significant given the smart home industry’s focus on their commercial devices predominantly using audio as the user interface method (e.g. Amazon’s Alexa and Google Home). It could be that much of this research is private rather than public. Either way it appears that there is a significant research contribution to be made in this area.

Location data is featured in the input data types from wearable devices and also motion tracking sensors. Mobile phones would be an obvious choice for further research in tracking people around smart spaces, but this may fall into a category with video where there are privacy concerns for the user. All these input data types could be categorised as obtrusive (mobile phone data, live video, audio) and unobtrusive (appliance and device data, motion sensors, power usage), which should be given consideration when conducting research into this area.

The mean prediction accuracy for all results is 79.19%, which could be used as a broad benchmark for current smart home behaviour prediction research.

4.2. How is behaviour modelled?

Modelling behavioural data can be done with several different types of data structures of increasing complexity that accordingly can reflect different levels of complexity within the behaviours. In order of complexity we have seen in the literature:

Graphs are particularly apt for modelling sequences, and inferring which element in a sequence might be next. An example of how a graph or graph like data structure can be used for sequence prediction is detailed in Fig. 4. The vertices can be mapped to each known event in a specific use case, and directed graphs can model the arrow of time. Finally, the edges not only connect the vertices, or events, in a logical way but can also have probability values attached to them that can be dynamically updated. For example, when at point A probabilities for each edge route to points B, D, and E become available (summing to 1). How these probabilities are calculated is determined by the prediction algorithm, which is often a function of the state change from one event to another based on previously observed data for these states.

4.3. What are the algorithms used for behaviour prediction?

A visual breakdown of algorithms into categories is shown in Figure 3. The best results for prediction accuracy are being achieved with machine learning algorithms or Bayesian networks, with the highest accuracy recorded in the review papers using Hadoop to process massive datasets that are then used to train a back propagation neural network variant (BPNN). The amount of training data available often impacts prediction accuracy in machine learning models, so this will have been a factor in the results. The highest prediction accuracies are also currently achieved by predicting simple behaviours such as a person moving from one location to another [63], rather than complex multi-occupancy behaviours. The most common algorithms for predicting human behaviour are variations of well known machine learning algorithms (NN, RL, Q-Learning, RNN/LSTM), and variations of LeZi, CRAFFT, and SPEED. All of these algorithms take different approaches to solving a fundamental problem, which is sequence prediction. If we consider a sequence of events or actions, represented by the stochastic process X = { x 1 } , the sequence prediction problem can be stated as follows. Given a sequence of events { x 1 , x 2 , … , x n } , what is x n + 1 ? Unsupervised learning algorithms do not feature in the literature but they could be used to infer next behaviours through clustering of the event data. Alternatively, they could be used as an intermediary step to enhance prediction by filtering out less likely next events e.g. by grouping events into time-windows, before then passing the reduced set of labels to a prediction model or algorithm.

OPEN IN VIEWER

Fig. 3.

Sankey diagram of prediction algorithms in review papers.

PGMs combine two features which make them ideal for behaviour prediction. The graph data structure, which can easily be used to model actions and state changes, combined with probability calculations on the edges which can be used for prediction values [7,8]. Probability is particularly suited for a stochastic process like behaviour as it can account for spontaneity and randomness, it can have multiple possible options with various levels of confidence, and can also be calculated with contextual factors taken into account. Additionally, it can also be updated in real time as new data is observed, so models deployed to smart environments could update themselves to adapt to individual occupant’s behaviours. PGMs generally fall into two categories: Markov model variants, and Bayesian model variants.

OPEN IN VIEWER

Fig. 4.

Simple probabilistic graph example.

4.5. Machine Learning (ML)

Machine Learning (ML) algorithms produce some of the best results found in this review for sequence prediction. Neural Networks (BPNN, ANN) and hybrid Bayesian/Neural Networks (DBN-ANN), Recurrent Neural Networks (RNN) which are particularly adept at time-series prediction [39], Reinforcement Learning (Q-Learning) which rewards correct predictions, and finally Support Vector Machine (SVM) which uses regression. ML algorithms have found their way into every corner of computer science, particularly for classification and prediction (inference) problems. To better understand why they are good for human behaviour prediction, or sequence prediction, it is easier to discuss them as higher level categories: Regression, Neural Networks (NN), Reinforcement Learning (RL), Long-Short Term Memory (LSTM), and Gated Recurrent Unit (GRU).

4.6. Statistical and trend analysis

Statistical and trend analysis are good methods for macro level behaviours over longer timeframes. They can calculate the likelihood that events might occur on monthly or annual time scales, rather than the highly focussed immediate activity predictions of ML or PGM algorithms. This type of analysis is often called forecasting, and could be used in smart environments in conjunction with other prediction methods, to take a longer term look into the likelihood of future events.

4.7. Other notable algorithms

There are other algorithms reported in the literature, that are either unique in their approach, or have been overtaken by developments in the three main categories outlined above.

4.8. Scenarios for behaviour prediction

The majority of high level use cases for behaviour prediction in smart environments are for Ambient Assistive Living (AAL), a subfield of Ambient Intelligence (AmI) – a still theoretical advanced smart environment with seamlessly embedded artificial intelligence. However, we are seeing the beginnings of these advanced environments with the use of smart technology in homes and assistive living settings. Assistive living environments are usually healthcare facilities or care homes, and use cases focus on Activities of Daily Living (ADL) and how to predict these events so that they can be preempted, and therefore assisted.

Simple use cases from the literature are systems that know when certain appliances or devices are going to be used, and can activate them instead of the occupant having to frequently do so [42]. Location prediction also features for determining where an occupant might move next in a smart home – which produced the highest prediction accuracy of all the review papers seen [63]. More complex use cases focus on anomaly detection in predicted behaviour. If a pattern of behaviour deviates from the predicted pattern an alarm can be raised that something unusual might be happening [41]; which can be particularly useful when passively monitoring the elderly.

The current research understanding, knowledge, and technology for computational human behaviour prediction is still underdeveloped and the use cases found in the literature reflect that. Basic scenarios are used as a proof of concept while we unravel the structure of predicting patterns that can become almost infinitely complex. One of the most fundamental questions that these use cases should address is: Why is behaviour prediction important for smart environments? The answer to this is that predictive systems are an evolution from the reactive systems that our current technology is capable of. With predictive systems there is no need to wait for a command to be issued. Based on data already seen a system can make a decision for a person. This cognitive offloading then frees the occupant of a smart environment to focus on other tasks; whether highly important ones such as in healthcare and professional settings, or more leisure and entertainment focussed at home. We have seen similar cognitive offloading happen as a result of technological advances in the past with conventional home appliances, computers, and domestic white goods that free people to focus their energy and attention on less mundane tasks. Human behaviour prediction capabilities for smart devices will take care of tasks before they occur to us; control power usage, IoT devices, and entertainment; and eventually even control vehicles and macro scale smart city systems [20] [6,54,64].

The use cases found in the literature focus on domestic smart home settings, healthcare, and care homes. However, it is likely that this research could be applied to many different types of settings – from vehicles to smart cities – using the same theoretical knowledge of sequence prediction and pattern recognition. Future research could investigate more complex use cases such as multi-occupant behaviour prediction where behaviours overlap, massive scale behaviour prediction like that found in crowds at sporting events, or industrial and commercial use cases for improving safety. Examples of these can be found by looking at the Ambient Intelligence use cases in the report by the Information Society Technologies Advisory Group (ISTAG) from 2001 [21], which are still relevant today.

4.9. Further research

Several opportunities for further research have been identified by this systematic literature review. These include:

Using different types of data input, such as video and/or audio, which is surprisingly missing from any of the research

4.10. Centres of research

Based on the number of papers per affiliation in the review data the centres of research for behaviour prediction computation are Washington State University, The University of Texas, and Nottingham Trent University. However, several of the datasets produced are from MIT, which may not have a focus on behaviour prediction, but does have a significant amount of research in smart environments. The top researcher in this field is Diane J. Cook of Washington State University with a significant amount of named author papers, and a high number of citations for those papers.