A context-aware system in Internet of Things using modular Bayesian networks

Abstract

Recently, the concept of Internet of Things has widely proliferated to offer advanced connectivity between devices, systems, and services that continuously obtain enormous amounts of data from sensors. Recognizing context from the sensor data plays a crucial role in adding value to the raw sensor data. In this article, we propose a context-aware system through device-oriented modeling for the Internet of Things using modular Bayesian networks based on our previous study. A Bayesian network can handle flexibly the uncertain environments of frequent changes in device configuration, and the proposed system can enable us to adjust to the changing Internet of Things environment, making it more flexible. The main contribution of the article lies in the realization of the modular context-aware system with device-oriented modeling of Bayesian networks in smart home and the verification of the usability through a subjective test with 116 people. In addition, we evaluate the performance of the proposed system and show the reduction of time complexity using the real data. Compared to other methods such as decision tree and monolithic Bayesian network, the performance improvement is statistically significant according to t-test.

Keywords

Context-aware services modular Bayesian networks Internet of Things smart TV

Introduction

Context-awareness is considered as the core feature of ubiquitous and pervasive computing systems which aim to provide convenient services to users who are in the contexts recognized. During the last two decades, the context-aware computing allows us to store context information linked to sensor data so that the interpretation can be done easily and more meaningfully.¹ In this situation, collecting and analyzing sensor data from all the sources were possible and feasible due to the manageable number of sensors. In contrast, Internet of Things (IoT) envisions an era where billions of sensors are connected to the Internet, where it is not feasible to process all the data collected by those sensors. Previous solutions dealt with different aspects in the IoT, such as device management, interoperability, platform portability, security, privacy, and context-awareness. However, context-awareness will become even more important in deciding what data need to be processed.

The main contribution of the article is to develop a context-aware system that transforms the raw data into knowledge by collecting, modeling, and reasoning the context to consolidate the sensor data together to infer new knowledge for the IoT. The system can achieve this goal by context-awareness with device-oriented modularization. To implement it, we highlight smart TV that is the next-generation television technology. It can interact with several devices in the house and work without battery limitation. Bayesian network (BN) is one of the powerful techniques for context-awareness in the uncertain and incomplete situation with a variety of devices such as smartphone, TV, embedded sensor, and air-conditioner. As the devices can be various from site to site and can be removed and added, we modularize the BN constructed independently with respect to its device and function.

We also develop a simulator for the IoT using the Unity3D tool which consists of six devices: smartphone, smart TV, air-conditioner, light, humidifier, and air cleaner. The superiority of the proposed system is confirmed by comparing the performance with real data, and the t-test is conducted for verifying the statistical significance of the results. In addition, we perform the System Usability Scale (SUS) test with 116 subjects to show the usefulness of the system. We already demonstrated that the modular approach to designing BNs was successful for landmark detection from mobile log data,² and the modular Bayesian network (MBN) could be improved with expert knowledge to develop a service robot.³ We have been investigating the power of probabilistic models in several areas and trying to solve the problem of scalability. This article develops another method of designing a modular context-aware system based on device-oriented modeling of BNs. We apply this method for smart home as one of the real problems in the IoT environment and evaluate the usefulness of the system with large number of subjects.

The rest of the article is organized as follows. Section “Related works” presents the related works for context-aware systems. Section “Context-aware system with smart TV for IoT” describes in details the context-aware system using a MBN. Section “Experiments” shows the experiments conducted to confirm the usefulness of the system.

Related works

Smart TV in IoT

IoT is a concept in which objects and people are provided with unique identifiers and the ability to transfer data over a network without requiring human-to-human or computer-to-human intervention. A number of sensors deployed around the world are growing at a rapid pace, where sensors generate enormous amounts of data and the devices to generate them are not static. We need to understand the meaning of data in order to provide appropriate IoT services.

Smart TV includes a computer and Internet access with high-quality screen and various sensors. In addition, it is possible to communicate among other appliances such as smartphone, air-conditioner, and humidifier through the Internet. Smart TV has been enabled to control their interlocking devices. With improving performance of smart TV, the trend of service is to provide greater personalization in IoT. Yusufov and Kornilov⁴ defined the roles of smart TV in IoT: information storage, visualization device, interaction point, data processor, and data source. Several researchers presented smart TV as the central point in IoT because it can store the collected information from sensors and process the context-aware system. Wu et al.⁵ implemented cognitive IoT. In the study, the smart TV is one of the appliances to collect data and provide the optimal service. Smart TV can be one of the devices to implement the framework of IoT.

Context-aware service

Context is all the information related to the interactions between user and applications.⁶ Such interactions are becoming important as research in pervasive computing progresses. Context-aware services are focused on the core technique for the IoT. Table 1 summarizes the works related to context-aware services in various domains and shows the contexts used. In the mobile robot domain, the context includes the input from the robot’s sensors. Park and Cho³ designed MBNs using domain knowledge to provide robot services at home. They inferred user states using the contexts of the user and environment. Similarly, Lacey and MacNamara⁷ proposed services using context-awareness to decide robot movement directions for the blind persons. They analyzed the information provided by sensors to infer the structure of the route using BNs.

Table 1.

Related works on context-aware services.

Domain	Author	Context			Proposed method	Service
Domain	Author	User	Env.	Device	Proposed method	Service
Robot	Park and Cho³	√	√		Modular Bayesian networks	Home service robot
Robot	Lacey and MacNamara⁷		√	√	Bayesian networks	Service robot for elderly blind
Smartphone	Yu et al.⁸	√		√	Naive Bayes classifier + rule-based method	Media recommendation
Smartphone	Herrmann et al.⁹			√	Rule-based method	Power management
Smart environment	Danninger et al.¹⁰	√	√		Rule-based method	Meeting room and schedule management
	Paganelli and Giuli¹¹	√	√		Ontology + rule-based method	Home service for elderly people
	Gu et al.¹²	√	√		Ontology + rule-based method	Inference of user location in building
	Fu et al.¹³	√	√	√	Ontology + rule-based method	Device management depending on user

Mobile phones have the advantage in that it is easy to collect context data with various sensors. Context-awareness in mobile phones usually uses low-level contexts. For example, Yu et al.⁸ proposed a media recommendation service. Their research described the need to recommend the right contents in the right form to the right person. They used a naïve Bayes classifier to infer user context and a rule-based method to determine the state of smartphone. They used the inference results to recommend contents to users, considering their current situation. Since a number of applications use sensors in mobile phones, power management is an issue. Herrmann et al.⁹ proposed a management method in mobile systems that defined the power model for each sensor, such as GPS, accelerometer, and Wi-Fi.

Context-awareness in the smart environment collects context data using the sensors of installed devices to provide services to users. For example, Danninger et al.¹⁰ proposed a method for managing meeting room and schedule. Paganelli and Giuli¹¹ presented the ontology for elderly people at home, such as patient state, home state, and emergency alarm. Their situation-aware strategy used the ontology to infer needed services using predefined rules. Gu et al.¹² defined ontology to infer user location. Fu et al.¹³ proposed device management services that depended on changes in user location with XML and a rule-based method. These methods using ontology or XML have difficulties to respond flexibly to uncertain situations and require a large amount of rules. It is also problematic that user states are simply reflected by pre-entered data in the service generation.

Katasonov et al.¹⁴ proposed the middleware for agent-based context-awareness in IoT. They believed that tasks of automatic integration, orchestration, and composition of complex systems on the IoT will be impossible in a centralized manner due to the scalability. Terziyan et al.¹⁵ proposed the middleware for smart road environment, aiming to govern seamless interoperation of devices, services, and humans. Badii et al.¹⁶ proposed the context-awareness framework for intelligent networked embedded systems. They defined that the context is three types: application, device, and semantics. They were integrated for reasoning higher-level context. These previous works proposed the framework or middleware and proved the necessity of integration of contexts. However, previous works did neither apply the proposed systems to real problems in IoT nor evaluate the usefulness of them.

BN

This article uses Bayesian probabilistic model to provide the services in IoT environment effectively. BN can infer in uncertain or incomplete situations. It is important to provide services in IoT because it has the numerous cases of incomplete situations. For instance, the device does not work or the state of the device changes. The system should provide the services effectively under changing situations.

BNs are models to express large probability distributions with relatively small cost in statistical mechanics. The structure is a directed acyclic graph (DAG) that represents the link (arc) relationship of each node and includes conditional probability tables (CPT). The model assumes that all nodes are independent of each other. A conditional probability distribution of variable A can be represented as $P (A | pa (A))$ , where $pa (A)$ denotes the set of parent variables of variable A, $U = A_{1}, A_{2}, \dots, A_{n}$ is a set of nodes, and the joint probability distribution is computed by the chain rule, as in equation (1)

\begin{array}{l} P (U) = P (A_{1}, A_{2}, \dots, A_{n}) \\ = P (A_{1}) P (A_{2} | A_{1}) P (A_{3} | A_{1}, A_{2}) \dots P (A_{n} | A_{1}, A_{2}, \dots, A_{n - 1}) \\ = \prod_{i = 1}^{n} P (A_{i} | p a (A_{i})) \end{array}

(1)

The training of BN consists of two components: structure and parameters. To identify the structure, the most frequently used algorithm is to learn it from the data. To obtain the parameters, maximum likelihood estimation based on real data is the most generally used. This approach requires sufficient data for learning the structure and parameters of the network. The other approach is to construct it with expert’s domain knowledge. This method can identify the structure and set parameters based on such knowledge, which is useful when there is insufficient data for training in the domain. Chen and Pollino¹⁷ investigated a design method of the BN for an environment problem. Ashcroft¹⁸ proposed an analysis method to solve problems between employee and employer. Hwang and Cho² proposed a MBN model for landmark detection from a mobile life log. Their method analyzed mobile log data effectively and extracted semantic information and memory landmarks. Zhu and Sheng¹⁹ proposed a method to recognize complex human daily activities for robot-assisted living systems. As these examples illustrate, BN techniques are promising to solve the problems in various domains.

The time complexity of BN can be calculated using the Lauritzen–Speifelhalter (LS) algorithm,²⁰ as given in equation (2), where n is the number of nodes, k is the maximum number of parents for each node, r is the number of values for each node, and w is the maximum clique

CMPX = O (k^{3} n^{k} + w n^{2} + (w r^{w} + r^{w}) n)

(2)

Context-aware system with smart TV for IoT

In our previous work, we proposed a probabilistic modeling method for context-aware service in smart TV.²¹ In this article, we extend the previous system to a smart TV–based context-aware system for IoT using MBN. This article includes more detailed information on how to design the BN in accordance with the complicated situations in IoT environment.

Figure 1 illustrates the proposed system. The types of context in the system are defined as user, user profile, environment, and state of devices. These contexts are collected using the sensors of devices in IoT.

Definition 1 (user): $u (t) \in U$ is the information in the set U of the user u which is pre-entered in the smart TV at current time t.

Definition 2 (user profile): $p (u) \in P$ is the profile of the current user in the set of user profile P.

Definition 3 (state of device): $s (d, t) \in S$ is the information of the current time t in the state of the devices $d \in D$ .

Definition 4 (environment information): The system includes the variable e, which represents the environment. The variable $e_{t} \in E$ is the information obtained by the sensors of the smart TV and the Internet connection.

Figure 1.

System architecture.

These contexts are collected using the device sensors in IoT. The context of the user is composed of the user states such as user’s movement, user’s position. The user profile is pre-registered information that does not change once it is entered, which is used to infer the user preference. The environment is a part of the context that affects the user and the devices in IoT. For instance, the light in the living room affects the smart TV’s brightness control and the user’s vision. The system defines this information as the context of the environment. To decide the service of the devices, the proposed system needs to know its current state, as well.

Obtained context is composed as a message $Msg = (u (t), p (u), s (t), e_{t})$ , which is preprocessed and converted into XML format. The BN requires discretization of continuous variables to reduce the complexity. Context-awareness selects the service and network considering the initial condition of each service, the user request or situation. Then, it infers the user state and the optimal service using each network modules and integrates the result used for providing the context services in IoT.

MBN and selective inference

The inference model in IoT should handle the state-changing problem and the model-expending problem, because changing requirements and dynamic environments are common for context-aware applications in IoT environment. As it becomes increasingly complex, the traditional solutions using a single model for context-awareness to manage and control devices reach their limits and pose a need for modular approach. When the new contexts from new devices are added, they have to reconsider all the contexts in a single model; the proposed system does not take into account of all the contexts at once. We just consider the module associated with the new contexts and decide the conditional relationship between the new and the existing contexts. In this way, we can prevent the vain efforts when the system needs extension. Figure 2 shows the usefulness of extension in the MBN, where the communication link means the connection between the devices over the network in real world. In this figure, the probable values in white nodes represent the information that needs to adjust the relationship when the new context is added. The improvable values in black nodes represent the information that does not need to adjust the relationship because the new values do not affect to its modules.

Figure 2.

The usefulness of extension in modular Bayesian network.

We propose a MBN with selective inference. MBN consists of multiple BN units (BN modules) connected with other units according to their service relationships. The selective inference allows us to use the necessary modules only to offer service instead of all modules. This method prevents from providing the useless services considering device which is neither on nor installed. BN, BN modules, and MBN are defined as follows:

Definition 5 (BN): BN is a probabilistic graphical model that represents casual relationships between random variables. BN consists of values V and edges $E = (V_{i}, V_{j})$ , and CPT $P (V)$ .

Definition 6 (BN module): A BN module $M_{i} = (G_{i}, P_{i})$ is a BN represented by $G_{i} = (V_{i}, E_{i})$ , where V_i’s are variables and E_i’s are edges, and P_i is the conditional probability.

Definition 7 (MBN): MBN consists of a 2-tuple (M, D), where M represents the BN modules and D indicates the device (see Figure 2).

Service analysis

An important problem of MBN is to decide the criterion to split up the modules. We classify the type of the services by devices because each device in IoT environment can communicate with others and it can be separated based on the service utilizing each device. Table 2 shows the service definition of each device and the initial conditions needed to provide possible services. In this table, the initial condition means when the service is stated because all systems need the start point of executing it.

Table 2.

Service and initial condition of each device.

Devices	Categories	Services	Conditions
Smart TV	Functions	Brightness and contrast	Change channel, turn on TV
		Sound effect	Change channel after a minute, turn on TV
		Volume	Change channel
	Personalized UI	Program recommendation	Recommend button
		Personalized function menu setting	Function button
		Personalized control menu setting	Control menu button
	Convenience function	Child protection	Child, change channel
		Power saving	User absent
		Auto-record	User absent
Smartphone	Personalized UI	Personalized function menu setting	Turn on smartphone
Smartphone	Convenience function	Schedule alarm	Existence of schedule
Humidifier	Environment	Humidity setting	User is located
Air-conditioner	Environment	Temperature setting	User is located
Air-conditioner	Environment	Air-cleaning	User is located

Smart TV has three categories of service with reference to the product guidebook of Samsung, LG, and Sony: function control, personalized user interface (UI), and convenience. TV function control services include brightness, contrast, sound effect, and volume controls. The services can control the basic functions of smart TV. Personalized UI provides the user with individualized UI considering user preference. Convenience service provides additional services using an interlocking device as well as schedule alarm, energy savings function, protections for children, and so on.

Smartphone can provide two services: personalized UI and schedule alarm. The personalized UI service in smartphone is inherited from that in smart TV. Schedule alarm service notices the schedule to user considering the importance and the relationship of it. To provide the convenience state of environment, air-conditioner and humidifier provide the optimal temperature setting and humidity setting service.

Figure 3 shows the definition of the BN modules, where the white boxes represent the relationship between service and device. The BN module is illustrated by the light gray and the dark gray boxes. The specific BN modules represented in light gray are implemented and used for a specific service only. The dark gray boxes mean the common BN modules, which is used for two or more services. For example, the smartphone and the smart TV can offer the function menu setting service and the color, font, and function menu are provided by BN modules.

Figure 3.

The definition of Bayesian network modules for service decision.

BN implementation

The service and domain analysis are to define the criterion to modularize the BN. This section presents our design of the lowest domain.²² Each module is designed in four steps: identification of input values, node design, structure design, and parameter design.

In the first step, the input values are identified through service and domain analyses. The input values obtained from each device consist of three factors: user, environment, and the state of device. The user input values are based on recognizing the current user state and preference using sensors in the smart TV and smartphone, such as age, gender, conversation, and clothes. The state of device input values is obtained from the sensors or the Internet connected. These values represent the genre of the current channel and the current state of the setting of devices. The input values of the environment affect the device and the user. These include temperature, humidity, weather, and brightness in the room. Table 3 shows the defined input values.

Table 3.

Description of input values.

Type	Device	Input value	Method of obtaining the value
State of device	Smart TV	Turn on/off	Camera input
		Genre	Web connection
		White level	Image analysis
		Black level	Image analysis
	Smartphone	Turn on/off	Cell phone connection
	Light	Color temperature	Light connection
	Light	Brightness	Light connection
	Air-conditioner	Temperature setting	Air-conditioner connection
	Humidifier	Humidity setting	Humidifier connection
User	Smart TV	User ID	Camera input
		Age	User input
		Position	Camera input
		Clothes	Camera input
		Movement	Camera input
		Conversation	Microphone input
		Air-conditioning preference	User input
		Heating preference	User input
		Brightness sensitivity	User input
		Ear dysfunction	User input
		Earphone usage	User input
	Smartphone	Schedule	User log data
		Call log	Phone log data
		Contact information	User input
Environment	Smart TV, smartphone	Time and date	Web connection
		Weather	Web connection
		Noise	Audio sensor
	Light	Brightness in room	Illumination sensor
	Humidifier	Indoor humidity	humidity sensor
	Air-condition	Indoor temperature	Temperature sensor
	Air-condition	Indoor air-condition	Air-condition sensor

In the second step, we have to identify the nodes and categorize them as query, input, or hidden nodes, which are constructed in a hierarchical structure. Each module is designed to consider the relationships of the nodes. Specifically, the input nodes influence hidden nodes, hidden nodes influence query nodes, and some important input nodes, like light or time, directly influence query nodes. The reason for using this hierarchical structure is to reduce the complexity of BN. The time complexity of inferring the BN is strongly influenced by the number of parent nodes. We use the hidden node to limit the maximum number of parent nodes as two, because the size of the CPT increases proportionally to this number. Figure 4 shows an example of a hidden node. Before using the hidden node, the sizes of the CPT for “light,”“sunny,” and “cloudy” were 8, for “time” was 28, and for “brightness” and “contrast” were 2. Therefore, the total size of the CPT was 58. However, after using the hidden nodes, the sizes of the CPT tables for “light,”“sunny,” and “cloudy” were 4, respectively, for “time” was 14, and for “brightness” and “contrast” were 2. The sizes of the hidden nodes of “brightness in room” and “weather” were 4. Therefore, the total size of the CPT was reduced to 38. We have confirmed the decrease of the CPT size using hidden nodes.

Figure 4.

An example of hidden nodes. The direct connections between the causal relationships are simplified by the hidden nodes of “Bright in room” and “Weather.”

We calculated the time complexity of the BN using equation (2). We have replaced the maximum clique size w with k, since the clique size is proportional to the size of the parent, and k is 2. We have designed the node to have not more than three parent nodes. Therefore, equation (2) is transformed to equation (3)

CMPX' = O (2^{3} n^{2} + 2 n^{2} + (2 r^{2} + r^{2}) n)

(3)

Next, we design a module structure using domain knowledge that has a direction between nodes. Basically, nodes in a BN have straightforward relationships from the causal node to the result node. However, the network is designed to use the reverse direction such that we infer the probability of the query node given the input node. This method makes it simple to design the module structure.

Finally, we design the node parameters according to the previous studies and statistical data. For example, the parameter of the user state refers to the book of TV living which is based on Gauntlett and Hill’s²³ investigation of the behavior of people in front of TV. The designed network should be evaluated for its performance in terms of whether it is satisfactory or not. Jakeman et al.²⁴ proposed to measure the performance of a BN using domain knowledge, which can be evaluated through scenarios. We have evaluated the performance of services. Figure 5 illustrates the user state BN using GeNie. The query nodes are represented as black boxes, hidden nodes as gray boxes, and input nodes as white boxes. Table 4 details each node in the user state network.

Figure 5.

User state BN. Six states of user’s activity are inferred from various low-level evidences via three hidden nodes of “sleeping,”“calling,” and “exercising” possibilities.

Table 4.

The nodes of user state BN.

Type	Name	State	Description
Input node	User position	{Lying, sitting, standing}	Possible positions depending on user state
	User movements	{Lot, normal, few}	Number of movements depending on user state
	User clothes	{Sport, outdoor, indoor, sleep}	Particular clothes depending on user state
	Cell phone	{Yes, no}	State of user’s cell phone
	Day of the week	{Weekday, weekend}	User state depending on day
	Light in the room	{Yes, no}	Brightness depending on light
	Sport	{Yes, no}	User state depending on genre
	Education	{Yes, no}	User state depending on genre
	Activity time	{Yes, no}	Activity over time
	Sleep time	{Yes, no}
	Meal time	{Yes, no}
Hidden node	Sleeping possibility	{High, normal, low}	Hidden node of possibility of user state
	Calling possibility	{High, normal, low}
	Exercising possibility	{High, normal, low}
Result node	Sleeping	{Yes, no}	Probability of inference results of each state
	Calling	{Yes, no}
	Exercising	{Yes, no}
	Studying	{Yes, no}
	Having a meal	{Yes, no}
	Watching TV	{Yes, no}

This reusability is an advantage of network modularization. Therefore, the inference complexity of MBNs is given in equation (4). The number of values of n is $n / d$ , where d is the number of modules

CMPX ″ = O (2^{3} \frac{n^{2}}{d^{2}} + 2 \frac{n^{2}}{d^{2}} + (2 r^{2} + r^{2}) \frac{n}{d})

(4)

We have implemented 23 BN modules based on the service analysis in section “Service analysis.”Table 5 shows the results of BN modules and describes the functions of BNs.

Table 5.

The results of BN modules.

Name	Result	Description
User state	{Sleeping, calling, exercising, studying, having a meal, watching TV}	Consideration of user state for offering the service
Brightness and contrast	{Brightness up, brightness down, contrast up, contrast down}	Management of brightness and contrast in smart TV
Sound effect	{Surround, equalizer, voice emphasis}	Setting optimal sound effect
Volume	{Up, maintenance, down}	Setting optimal volume
Genre	{News, drama, sports, documentary, travel, movie, food, music, comedy, talk show, entertainment, amusement}	Program genre recommendation using user profile
Position	{Up, center, right, left}	Setting menu position
Color	{Red, blue, pink, green, gray}	Setting menu color
Font	{Graphic, min-style, gothic, gulim}	Setting optimal font type
Function menu	{Screen, sound, channel, media, external-input, self-diagnosis, initializing}	Emphasizing the function menu button that the user uses at current time
Video menu	{Play, stop, pause, forward, backward}	Emphasizing the video menu button that the user uses at current time
Content lock	{Yes, no}	Locking the content considering user’s age
Energy saving	{Yes, no}	Saving energy of smart TV when the user is absent
Short time absent	{High, normal, low}	Prediction of time of user’s absence
Long time absent	{High, normal, low}	Prediction of time of user’s absence
Record	{Yes, no}	Selecting auto-record function
Fatigue	{High, normal, low}	Awareness of user’s fatigue
Important schedule	{Yes, no}	Awareness of importance of schedule
Emergency situation	{Yes, no}	Awareness of emergency situation
Requirement schedule	{Yes, no}	Selecting whether or not to alarm schedule
Relationship	{Close, normal, distant}	Friendship with user
Humidity	{High, normal, low}	Management indoor humidity
Temperature	{High, normal, low}	Management indoor temperature
Air-cleaner	{High, normal, low}	Management indoor air-condition

Service decision

The system needs to integrate for decision-making using the resulting MBN. First, when the user requires the service, the system checks the initial condition and the related modules of the service. If the device offered the service is not implemented, the system does not offer service or provides a part of the service.

Second, the system calculates the probability of the modules by evidence setting. When the system collects the context from the devices, it is the evidences of BN modules. The situation collected the evidence is in three types: collection failed, no state changed, and state changed. “Collection failed” is occurred when the device does not connect to the smart TV. If the state of the evidences is not changed as comparing from the previous state, the type of evidence is “no state changed.”“State changed” means that the evidence is collected without any problem.

Third, the system integrates the probability results of BN modules and controls the device considering the results. The probability results are produced when it is above the threshold. Algorithm 1 shows the pseudo code of service decision steps.

Algorithm 1. The pseudo code of service decision.
Input: service order, new evidence set $ε_{current}$
Output: service decision
Set_Priority_Module ( $Ψ_{R}, R$ )
$Ψ_{R}$ = Select_TargetServiceModule_by_Requirement
$R$ = Service requirement
repeat: Number of $Ψ_{R}$ = = 0
if ( $ε_{current}$ = = $\emptyset$ ) //Collection failure
Send_Message (“Do not offer the service”)
else if ( $\exists ε_{t}$ in Queue ≡ $ε_{current}$ ) // No changed state
${Ψ_{R}}_{out}$ = Load_PreviousProbability ( ${Ψ_{R}}_{out}$ )
else // Changed state
${Ψ_{R}}_{out}$ = Update_Probability_Module ( $Ψ_{R}, ε_{current}$ )
Save_InferenceResult ( ${Ψ_{R}}_{out}$ )
Integration_Probability_Module ( $Ψ_{R}, ε_{current}$ )
Control_device ( $Ψ_{R}, D$ )

Let us explain the process of the system with one scenario. A woman who is about 30 years old returns home. After she turns on the light in the living room, she changes her clothes and turns on the smart TV. She sits on a sofa and watches the TV. After a while, she takes her USB out of her bag and connects it to the TV. Smart TV recognizes the aerobic video file in the USB. She stands up in the middle of the living room and plays the video. She selects the video menu and pushes the forward button. The video is being played, while she is exercising.

The main objective of the personalized UI service in smart TV is to provide convenience to the user. It mainly considers the personal preferences of the user. In this situation, the smart TV provides personalized control of the menu service. This service uses three BNs in Figure 3: video menu, color, and position. The results of these BNs are integrated to provide the service. Figures 6 –9 show the results of each BN.

Figure 6.

Color BN.

Figure 7.

Position BN.

Figure 8.

Video menu BN at movie start.

Figure 9.

The change of video menu BN.

The probability of the pink node shows the highest score in the result of the color BN. This result is subject to the preferences of season and female. For the video menu BN, the probability of the state of pause is 0.23 and the state of forward is 0.21. This reflects the users’ general tendency. Specifically, we usually desire to skip early unnecessary parts of a movie, such as advertisements and the opening credits. In Figure 7, we show the inference of the position of the menu considering the location of the user. The menu is placed in the middle because the user sits on the sofa.

Figure 9 shows the results of the video menu BN when she pushes the forward button. This result reflects the general tendency of users who repeatedly push the forward button. In this scenario, the color BN and the location BN are not used because her other states do not change, so the previous results of the color BN and the position BN are reused without re-inference. Smart TV integrates the results and provides the personalized video menu service. This service occurs for the smart TV of the living room simulator when she pushes the forward button.

Experiments

To evaluate the performance of the services, we implement the IoT simulator using the Unity3D, which includes a man, a woman, a smart TV, an air-conditioner, a heater, and so on. In the simulator, we chose the scenarios that could realistically happen in a living room. The man and woman are the users of the smart TV, and they can show actions for the user state BN (sleeping, calling, exercising, studying, having a meal, and watching TV) in the simulator. Also, we can see provisions of services through changes on the screen of the smart TV and the states of the device. Figure 10 shows a snapshot of the living room simulator.

Figure 10.

Home simulator with IoT environment.

Time complexity

We want to confirm that the proposed system has lower time complexity than monolithic method and provides the services considering the changing states of devices in IoT. This section explains two comparative results to evaluate the usefulness of the proposed system.

First, we compare MHBN (modular BN using hidden nodes), MBN, OHBN (original BN using hidden nodes), and OBN (original BN using the LS algorithm). We have designed 26 networks that on average consisted of 3 query nodes, 4 hidden nodes, and 10 input nodes. We assume that the nodes have two states, and the maximum number of parents for each node is 3 when the network does not use the hidden nodes. Figure 11 shows the change of the complexity according to the number of modules. There is no change in the complexity of OBN and OHBN. These methods consist of only one module. The complexity of MHBN and MBN increases as the growth of the number of modules. As a result, the complexity of MHBN is lower than the other methods. Although the total number of nodes increases, setting a limit on the maximum number of parent nodes using hidden nodes effectively reduces the time complexity. Because this method reuses the previous modules when the user state is unchanged, the time complexity of the proposed system is reduced.

Figure 11.

The time complexity of Bayesian networks.

Second, we calculate the number of the useless nodes in the changing situation of IoT because the useless nodes cause some error or overhead in the probability calculation process. The implemented home simulator has five devices in the room: (1) smart TV, (2) smartphone, (3) air-conditioner, (4) humidifier, and (5) lamp.

Table 6 shows the number of the useless nodes in each situation that changes the state of devices like real world. Situation 1 represents that all devices are implemented in the room. In this situation, there is no difference of the number of useless nodes between monolithic BN and the proposed system. In case of situation 2 that does not connect the smartphone to smart TV, the number of useless nodes is 60. The numbers of nodes in fatigue BN, important schedule BN, emergency situation BN, requirement schedule BN, relationship BN, and user state BN are 14, 5, 9, 9, 18, and 5, respectively. In case of other situations that do not have more than one device, the proposed system has the lower number of useless nodes than the monolithic BN. The proposed system does not use the modules that are not implemented in the current situation before the probability calculation process. This state between the devices can be known through the Internet connection in IoT environment. Therefore, the proposed system is more suitable to IoT environment than the monolithic BN.

Table 6.

Number of useless nodes.

Situation	Devices					Number of useless nodes
Situation	1	2	3	4	5	Monolithic BN	Proposed method
1	O	O	O	O	O	0% (0/375)	0% (0/375)
2	O	×	O	O	O	16% (60/375)	1% (5/375)
3	O	O	×	O	O	7% (28/375)	1% (1/375)
4	O	O	O	×	O	7% (28/375)	1% (1/375)
5	O	O	O	O	×	6% (21/375)	1% (1/375)
6	O	O	×	×	×	21% (77/375)	1% (3/375)

Performance of service

In order to evaluate the inference performance, we compare the results of the proposed system, the monolithic BN, and the decision tree. Tables 7 –9 show the details of the classification confusion matrix of each method for inferring the user state: 0 (sleeping), 1 (calling), 2 (examining), 3 (studying), 4 (having a meal), and 5 (watching TV).

Table 7.

Confusion matrix of decision tree.

User state	0	1	2	3	4	5
0	312	0	25	39	24	0
1	0	400	0	0	0	0
2	28	0	310	9	23	31
3	0	3	1	396	0	0
4	65	0	22	60	209	44
5	22	0	7	0	40	331

Shading terms highlights the cases of correct classification.

Table 8.

Confusion matrix of monolithic Bayesian network.

User state	0	1	2	3	4	5
0	310	0	17	29	43	1
1	0	400	0	0	0	0
2	23	0	327	6	12	32
3	15	3	1	365	16	0
4	49	0	16	56	248	31
5	20	0	9	0	44	327

Shading terms highlights the cases of correct classification.

Table 9.

Confusion matrix of the proposed system.

User state	0	1	2	3	4	5
0	320	0	0	53	0	27
1	0	400	0	0	0	0
2	0	1	366	8	0	25
3	0	0	0	400	0	0
4	0	0	4	58	333	5
5	0	0	33	0	0	367

Shading terms highlights the cases of correct classification.

Situation 1 (calling) is correctly inferred for all the methods, because the system with smart TV can recognize the calling situation correctly when the user using the smartphone and the smartphone can communicate to smart TV. However, for the other situations, the proposed system has higher performance than the decision tree and the monolithic BN. Especially, for the activity of watching TV, the proposed system correctly recognizes the misclassified cases for sleeping and having meal of the other methods.

Figure 12 shows the average result of each method. The proposed system provides much better performance for all the three measures of precision, recall, and accuracy. In order to determine whether this difference is statistically significant or not, we conduct the paired t-test. The t-test results in 2.702 and our hypothesis of no improvement is rejected at 99.5% according to t-distribution table with the degrees of freedom as 5.

Figure 12.

The average result of precision, recall, and accuracy.

In addition, the quantitative performance of service decision of each service is evaluated using the input data from scenarios, as described in section “Service decision.” Each service is evaluated using 50 scenarios. We make the answer selected by the modules in the scenario and compare them with the inference results of the modules. Table 10 summarizes the performance including the number of test cases for each service. The average performance of service decision is 0.89. The results confirm that the performance is acceptable for every service.

Table 10.

Performance of service decision.

Devices	Categories	Services	Performance
Smart TV	Functions	Brightness and contrast	0.8 (12/15)
		Sound effect	0.93 (14/15)
		Volume	0.86 (13/15)
	Personalized UI	Program recommendation	0.93 (14/15)
		Personalized function menu setting	0.86 (13/15)
		Personalized control menu setting	1.00 (15/15)
	Convenience function	Child protection	1.00 (15/15)
		Power saving	0.93 (14/15)
		Auto-record	0.86 (13/15)
Smartphone	Personalized UI	Personalized function menu setting	0.86 (13/15)
Smartphone	Convenience function	Schedule alarm	0.8 (12/15)
Humidifier	Environment	Humidity setting	0.93 (14/15)
Air-conditioner	Environment	Temperature setting	0.93 (14/15)
Air-conditioner	Environment	Air-cleaning	0.93 (14/15)

Usability test

To evaluate the usability of the proposed system, we have requested the answers after we let users experience the system. For evaluation, 10 questions in SUS, which were proved a robust, reliable, and low-cost usability assessment tool, were used. SUS test of 116 subjects measures three aspects of the system: effectiveness (can users successfully achieve their objectives), efficiency (how much effort and resource are expended in achieving those objectives), and satisfaction (the experience satisfactory).²⁵ Subjects should give the answer of five degrees from “strongly disagree” to “strongly agree.”

Figure 13 provides the average score of each question for the 116 subjects. The scores for the odd-numbered questions are high, and those for the even-numbered questions are low, which is a desirable result for this test. The SUS provides a converted total test score, as shown in equation (5), with possible scores ranging from 0 to 100

\begin{array}{l} S U S r e s u l t = (o d d n u m b e r e d q u e s t i o n - 5) \\ \times 2.5 + (25 - e v e n n u m b e r e d q u e s t i o n) \times 2.5 \end{array}

(5)

Figure 13.

The scores of 116 subjects by questions.

Figure 14 shows the average test scores of the 116 subjects by gender and age. The result is a single score on a scale of 0–100, and our result shows the average score as 74.35. There is no absolute threshold to decide the superiority from the score, but according to the literature on the SUS performance,²⁵ this can be regarded as good score.

Figure 14.

Average score of SUS test.

Conclusion and future works

This paper proposed a system composed of MBNs for context-aware services in IoT environment. To reduce the time complexity, we limited the maximum number of parent nodes using hidden nodes. The modularized BN has the advantages of reducing the time complexity and reusing the modules considering the changing state of devices. To confirm the efficiency of the proposed system, we calculated the time complexity using the LS algorithm and compared the performance of the inference. In addition, we designed various scenarios to obtain realistic input data from simulations. We confirmed the performance of the services using these data along with subjective user satisfaction based on the SUS test.

For future works, we will implement the real-world environment using radio-frequency identification (RFID), smart TV, air-conditioner, and so on. The devices in the environment have various sensors and can obtain the data. The structure and parameter of BN modules are trained using them. In addition, the improvement of learning method is one of crucial components for good performance of the system. We will conduct image processing for a preprocessing step to identify user name, detect user position and user clothes, and so on. Also, speech recognition part is needed to understand user requests. Hopefully, the proposed system will be compared with the several alternative context-aware systems for IoT in the near future.

Footnotes

Academic Editor: Davide Brunelli

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea government (MSIP) (R0124-16-0002, Emotional Intelligence Technology to Infer Human Emotion and Carry on Dialogue Accordingly).

References

Perera

Zaslavsky

Christen

Context aware computing for the internet of things: a survey. IEEE Commun Surv Tut 2014; 16(1): 414–454.

Hwang

K-S

Cho

S-B.

Landmark detection from mobile life log using a modular Bayesian network model. Expert Syst Appl 2009; 36: 12065–12076.

Park

H-S

Cho

S-B.

A modular design of Bayesian networks using expert knowledge: context-aware home service robot. Expert Syst Appl 2012; 39: 2629–2642.

Yusufov

Kornilov

. Roles of smart TV in IoT-environments: a survey. In: Proceedings of the 3rd conference of open innovations association FRUCT and seminar on e-tourism, Petrozavodsk, Russia, 22–26 April 2013, pp.163–168. New York: IEEE.

Liao

LC.

Service-oriented smart-home architecture based on OSGi and mobile-agent technology. IEEE T Syst Man Cy C 2007; 37(2): 193–205.

Dey

AK.

Understanding and using context. Pers Ubiquit Comput 2001; 5(1): 4–7.

Lacey

MacNamara

Context-aware shared control of a robot mobility aid for the elderly blind. Int J Robot Res 2000; 19(11): 1054–1065.

Zhou

Zhang

. Supporting context-aware media recommendations for smart phones. IEEE Pervas Comput 2006; 5(3): 68–75.

Herrmann

Zappi

Rosing

. Context aware power management of mobile systems for sensing applications. In: Proceedings of the conference on information processing in sensor networks, Beijing, China, 16–20 April 2012, pp.1–5. New York: ACM.

10.

Danninger

Flaherty

Bernardin

. The connector: facilitating context-aware communication. In: Proceedings of the 7th international conference on multimodal interfaces, Taranto, 4–6 October 2005, pp.69–75. New York: ACM.

11.

Paganelli

Giuli

An ontology-based system for context-aware and configurable services to support home-based continuous care. IEEE T Inf Technol B 2011; 15(2): 324–333.

12.

Wang

Pung

. An ontology-based context model in intelligent environments. In: Proceedings of the communication networks and distributed systems modeling and simulation conference, San Diego, CA, 18–21 January 2004, pp.270–275. New York: IEEE.

13.

Chen

. A configurable context-aware simulator for smart home systems. In: Proceedings of the 6th international conference on pervasive computing and application, Port Elizabeth, South Africa, 26–28 October 2011, pp.39–44. New York: IEEE.

14.

Katasonov

Kaykova

Khriyenko

. Smart middleware for the internet of things. In: Proceedings of the ICINCO-ICSO ’08, Funchal, 11–15 May 2008, pp.169–178. Setubal: INSTICC Press.

15.

Terziyan

Kaykova

Zhovtobryukh

. UbiRoad: semantic middleware for context-aware smart road environments. In: Proceedings of the 5th international conference on internet and web application and services, Barcelona, 9–15 May 2010, pp.295–302. New York: IEEE.

16.

Badii

Crouch

Lallah

. A context-awareness framework for intelligent networked embedded systems. In: Proceedings of the 3rd international conference on advances in human-oriented and personalized mechanisms, technologies and services, Nice, 22–27 August 2010, pp.105–110. New York: IEEE.

17.

Chen

Pollino

CA.

Good practice in Bayesian network modelling. Environ Modell Softw 2010; 37: 134–145.

18.

Ashcroft

Bayesian networks in business analytics. In: Proceedings of the federated conference on computer science and information system, Wroclaw, 9–12 September 2012, pp.134–145. New York: IEEE.

19.

Zhu

Sheng

Realtime recognition of complex human daily activities using human motion and location data. IEEE T Biomed Eng 2012; 59(9): 2422–2430.

20.

Namasivayam

Prasanna

. Scalable parallel implementation of exact inference in Bayesian networks. In: Proceedings of the international conference on parallel and distributed systems, Minneapolis, MN, 12–15 July 2006, pp.8–15. New York: IEEE.

21.

Yang

Cho

S-B

. Probabilistic modeling for context-aware service in smart TV. In: Proceedings of the 2013 4th international conference on intelligent systems modelling and simulation (ISMS), Bangkok, Thailand, 29–31 January 2013, pp.78–83. New York: IEEE.

22.

Bryson

Stein

LA.

Modularity and design in reactive intelligence. In: Proceedings of the 17th international joint conference on artificial intelligence, Seattle, WA, 4–10 August 2001, pp.1115–1120. New York: ACM.

23.

Gauntlett

Hill

Chapter 2: television and everyday life. In: Gauntlett

Hill

(eds) TV living: television, culture, and everyday life. New York: Routledge, 1999, pp.21–51.

24.

Jakeman

Letcher

Norton

PJ.

Ten iterative steps in development and evaluation of environmental models. Environ Modell Softw 2006; 21(5): 602–614.

25.

Bangor

Kortum

Miller

JT.

An empirical evaluation of the system usability scale. Int J Hum-Comput Int 2006; 24: 574–594.

User state	0	1	2	3	4	5
0	312	0	25	39	24	0
1	0	400	0	0	0	0
2	28	0	310	9	23	31
3	0	3	1	396	0	0
4	65	0	22	60	209	44
5	22	0	7	0	40	331

User state	0	1	2	3	4	5
0	310	0	17	29	43	1
1	0	400	0	0	0	0
2	23	0	327	6	12	32
3	15	3	1	365	16	0
4	49	0	16	56	248	31
5	20	0	9	0	44	327

User state	0	1	2	3	4	5
0	312	0	25	39	24	0
1	0	400	0	0	0	0
2	28	0	310	9	23	31
3	0	3	1	396	0	0
4	65	0	22	60	209	44
5	22	0	7	0	40	331

User state	0	1	2	3	4	5
0	310	0	17	29	43	1
1	0	400	0	0	0	0
2	23	0	327	6	12	32
3	15	3	1	365	16	0
4	49	0	16	56	248	31
5	20	0	9	0	44	327

User state	0	1	2	3	4	5
0	312	0	25	39	24	0
1	0	400	0	0	0	0
2	28	0	310	9	23	31
3	0	3	1	396	0	0
4	65	0	22	60	209	44
5	22	0	7	0	40	331

User state	0	1	2	3	4	5
0	310	0	17	29	43	1
1	0	400	0	0	0	0
2	23	0	327	6	12	32
3	15	3	1	365	16	0
4	49	0	16	56	248	31
5	20	0	9	0	44	327