Abstract
With the recent proliferation of smart phones, they become useful tools to implement high-confidence cyber-physical systems. Among many applications, context sharing systems in mobile environment attract attention with the popularization of social media. Mobile context sharing systems can share more information than web-based social network services because they can use a variety of information from mobile sensors. To share high-level contexts such as activity, emotion, and user relationship, a user had to annotate them manually in previous works. This paper proposes a mobile context sharing system that can recognize high-level contexts automatically by using Bayesian networks based on mobile logs. We have developed a ContextViewer application which consists of a phonebook and a map browser to show the feasibility of the system. Experiments of evaluating Bayesian networks and performing the SUS test confirm that the proposed system is useful.
1. Introduction
Recent advancement of ubiquitous sensing, mobile and embedded computing, and wireless communication leads to a new system called cyber-physical systems (CPSs) [1]. They are combination of computation, networking, and physical dynamics where embedded devices are massively networked in order to sense, observe, and control the physical environment. Mobile devices like smart phones can be a good tool for this cyber-physical system since they include various sensors, work as a small computer, and communicate with other devices in wireless networks. Moreover, the number of smart phone users still has been growing dramatically.
On the other hand, a number of web-based social network services including Facebook (http://www.facebook.com/), MySpace (http://www.myspace.com/), and LinkedIn (https://www-linkedin-com.web.bisu.edu.cn/) have been deployed and got explosive popularity with the dissemination of the Internet and social media. Social network sites offer a new way to make and maintain relationships [2]. Because of popular use of smart phones, interest in mobile context sharing systems is increasing for helping social interaction like social network sites. Mobile devices have a variety of sensors like a GPS receiver and a Bluetooth sensor to catch the surrounding and personal contexts [3]. These sensors enable users to communicate with friends by sharing their photos, locations, and so on. Based on this feature, many applications are developed for social network services in mobile environment.
Though a number of mobile context sharing systems are introduced, they cannot generate user's high-level contexts such as activity and emotion automatically. These high-level contexts are more useful for various practical services than easily obtainable low-level information. Previous works should annotate them manually to collect and share this information. This manual annotation is time consuming and can produce incorrect information. Some methods infer high-level contexts based on mobile logs, but they are not adapted to context sharing systems practically. It is required for a system to use a way to infer high-level context considering characteristics of mobile computing in terms of uncertainty, capacity, and power.
This paper presents a system that can recognize users' high-level context based on information collected from mobile devices and develops an application that can share the context based on social relationship among users. The system collects mobile logs, infers high-level contexts with Bayesian networks (BNs), and realizes an application for a smart phone. This system uses a server-client model because a server can cope with the limitation of capacity and battery power of a smart phone. ContextViewer is a user interface for smart phone using the system, which consists of a phonebook and a map browser to effectively deliver user's context. As sharing users' contextual information more easily and comfortably, the system lets them know their friends' situations and facilitates more communication.
2. Related Works
2.1. Mobile Context Sharing
Sharing contextual information in mobile environment is a hot issue in building social relationships. Many researchers have studied how to construct sensor platform, collect and preprocess features, and design system. ContexPhone is a software platform for context-aware applications. This system acquires context data such as location (Cell ID and GPS) or phone usage from sensors. ContextContacts, one of the applications with this platform, lets users automatically represent and exchange context information [4]. SharedLife is introduced as a generic and reusable content-sharing framework. It collects data from a variety of sensors, stores the information as a set of semantic knowledge models for the user's digital memory, and enables the user to share these memories with others [5]. MyWorld is a context-aware and social networking application. It is focused on activity sharing, real-time location sharing, and media sharing between phonebook contacts [6]. In these previous works, they attempted to recognize and deliver only the low-level contexts. If a user wants to represent their high-level contexts such as activity and emotion, it is required to annotate them manually. The proposed system uses the high-level context recognition model to overcome this problem.
Nomatic is designed for creating content passively by focusing on status messages in instant-messaging (IM) clients as short customizable phrases like “at lunch”. Using these cue phrases and sensors, Nomatic recognizes user's activity, place, and other high-level information based on machine learning [7]. While this system supports indoor and laptop environments based on APs (Access Points), it does not cover outdoor and mobile environments sufficiently. Besides, when the system does not have sufficient status message, it is difficult to infer contexts. Santos et al. proposed a method to recognize user activities like walking, running, idle and resting, and presented context sharing applications for instant messenger service, hi5, and microblog service, twitter [8]. Their contexts, however, are very simple and restricted. With advancement of SNS (social network service), context sharing services have been proliferating, but most contextual information is restricted comparing to the high-level contexts in the proposed system. In the previous work, we made recognition models of user activity and emotion for context sharing application [9]. This paper considers additional high-level context of relationships between users and constructs mobile social network using the information. Social relationships between users and mobile social network allow the proposed context sharing system not to depend on manual setting any more.
2.2. Bayesian Network-Based Modeling and Recognition
Bayesian network has emerged as a powerful technique for handling uncertainty in complex domains [10]. It is a model of a joint probability distribution over a set of random variables. The Bayesian network is represented as a directed acyclic graph where nodes correspond to variables and arcs correspond to probabilistic dependencies between connected nodes
Equation (1) represents Bayesian network formally. B and
There are two approaches to identify structure and parameters of Bayesian network model. The first approach is learning model from the data. Availability of data depends on the problem domain, and learning is a better choice when we have a lot of data. There are well-known algorithms for learning structures and parameters of Bayesian networks. The K2 algorithm presented by Cooper and Herskovits is the most frequently used algorithm to learn the structure of Bayesian network [11], and maximum likelihood estimation is the most general method to learn parameters of Bayesian network based on statistics [12]. The second one is manual modeling based on domain knowledge, which is crucial in modeling Bayesian network. Experts identify the structure and set parameters based on their knowledge, which is very useful because we cannot obtain reliable data in many real-world problems.
Xu et al. designed a user preference Bayesian network model based on empirical analysis and domain knowledge and applied it to a mobile application [13]. Hong et al. modeled hierarchical Bayesian networks manually for mixed-initiative interaction of human and service robot [14]. A few groups presented a systematic procedure for Bayesian network modeling. Marcot et al. described a guideline for Bayesian network design and suggested it to apply to ecological modeling problem [15]. Laskey and Mahoney proposed a Bayesian network engineering method based on the spiral system lifecycle model of software engineering technique [16]. These works presented a general Bayesian network modeling approach and applied it to real-world problems. On the other hand, a few works used a concept of functional module to design Bayesian networks. Neil et al. presented a procedure to construct a large-scale Bayesian network by using the idiom meaning functional module [17]. Marengoni et al. designed Bayesian network for image understanding and used procedural approach based on functional modules to solve complex problems [18].
There have also been various attempts for recognition using Bayesian networks. Hwang and Cho used a modular Bayesian network model to detect significant events from user's mobile life logs [19]. Krause et al. used BNs to recognize user preferences. To provide smart services, they clustered log data collected on mobile and wearable sensors, discovered a context classifier that reflected a given user's preferences, and estimated the user's situation [20]. ConaMSN recognized various indoor contexts such as the level of stress, the type of emotion and activity with Bayesian networks and visualized them with a set of icons on a messenger application. The proposed system collected experimental data from smart phones so that we can show the usefulness of the proposed models in uncertain mobile environment [10] and adopted the manual modeling approach for high-level context recognition so that we can overcome the lack of reliable data for learning.
3. Proposed System
We have designed a context sharing system consisting of four modules and user interfaces as shown in Figure 1. Because of the limited resources of smart phone, this system adopts client-server model. When a mobile client connects a server, created is a thread that consists of a preprocessor, a recognizer, and a social network manager. After generating a thread interacted with the client, a log collector collects mobile logs which are low-level contexts from sensors such as GPS coordinates, and call logs. These logs are transported to a server through TCP/IP. A preprocessor generates meaningful logs from web data collected in a server as well as mobile logs collected in mobile clients. A recognizer infers user's high-level contexts of activity, emotion, and relationships between users from refined logs. High-level contexts and meaningful logs are stored in users' context database. A social network manager builds mobile social networks based on high-level contexts including user relationships. A service manager takes a role of presenting contexts to a user and requesting what a user wants to see to a server. When the server receives a request from a mobile client, the server sends contexts to a smart phone. ContextViewer is a user interface to provide a user with information effectively. It consists of two parts: a phonebook and a map browser.
System overview.
3.1. Log Collector
The log collector module continuously gathers available low-level information such as call logs, SMS logs, GPS (Global Positioning System) coordinates, device status, Bluetooth, and weather from web in a mobile device. Most of context information is sent in the predefined expiration time for each log, but web data are collected in server part to reduce the communication bandwidth and power of mobile device. The data are used to recognize the user's high-level contexts such as what the user is doing and how the user feels.
Call and SMS logs are collected when an event occurs. For example, whenever the user makes call, the smart phone records logs including phone number, call type (send/receive/miss), start time, and end time. The module sends GPS logs and Bluetooth logs to a server periodically. The GPS logs present the places where the user visited. Bluetooth sensor can discover devices which allow the mobile device to collect information on other Bluetooth devices carried by people nearby. Weather log is obtained from web based on location and time. If weather or temperature is changed, the module writes a new log. Table 1 shows the types of logs. The logs for several sources are sorted according to the time.
Collected mobile life logs.
3.2. Preprocessor
The preprocessor module is in charge of producing meaningful information from raw mobile logs. Preprocessing step includes correcting data and annotating places. Data correction is to correct wrong data because of no signal. When GPS signals are not available, it writes a GPS log which has a value of the latest coordinate received from satellites. Because one of the purposes of the proposed system is to share information in real time, this approach is not used, though it is possible to infer locations by using previous coordinates and next coordinates. Place is labeled by the module. GPS logs, Bluetooth logs, and time are important traits for this labeling. If a GPS device in a smart phone detects coordinates, a user locates in outdoor. In outdoor environment, the name of a place can be labeled by matching near GPS latitude and longitude with a location in the predefined location table. Examples of matching a GPS coordinate to a place in Yonsei University are shown in Table 2. Bluetooth logs can be important clues to infer a location in indoor environment. For example, if a Bluetooth receiver catches Bluetooth ids of coworkers' computers, the user is in his workplace. Locations annotated are used to infer high-level contexts.
Examples of matching a GPS coordinate to a place in Yonsei University.
3.3. Context Recognizer
The context recognizer models high-level contextual information to recognize. Here, high-level contexts include user activity, emotion, and relationship between users. Table 3 shows the available values for each context. As data from sensors on a smart phone are often missing, uncertain, and incomplete, it is required to deal with uncertainty and missing values. BNs can address such problems by providing a robust inference based on probability. In this paper, the recognizer uses BN to model and recognize user's high-level contexts. We have designed Bayesian network models based on domain knowledge for general users. Modeling process includes designing the structure and parameters. For inference, we have used a well-known BN library SMILE (http://genie.sis.pitt.edu/).
The available values for high-level contexts used.
In order to recognize activity, BNs consist of five factors: mobile device status, spatial, temporal, environmental, and social factors. These are shown in Table 4 together with the corresponding variables. The recognizer calculates the probability of each activity at present based on these modules. Each activity is differently influenced on each module. For example, a temporal factor module has an effect on regular actions such as “meal” and “sleep,” but an environmental factor module does not. “Sport” is sensitive to weather that is an environmental factor module. Figure 2 shows an example of BN which is a model to infer “Sport” activity.
Five factors and the corresponding variables of activity recognition BNs.
A Bayesian network to infer “Sport” in the recognizer.
BNs to infer emotion use the result of activity inference BNs since activity has an influence on user's emotion directly. Inferred results of BNs are represented as axes of arousal and valence. It is based on arousal-valence emotion value defined as Table 5. Figure 3 shows BNs designed to recognize emotion. The probability of arousal and valence indicates the nearest feeling in Table 5. Figure 4 illustrates the BN to infer the relationship between users. It infers private and work relationships between two users.
Arousal-valence value and the corresponding emotion.
A Bayesian network to infer emotion (Inferred result corresponds to “Excited”).
A Bayesian network to infer the relationship between users.
Figure 4 provides a Bayesian network to infer the relationship between two users. It uses the activity and emotion inferred as well as other mobile logs of call, SMS, common schedule, and proximity based on Bluetooth information and user activities. If two users are searched by the device of the other user and their activities are private, it is set as P_Proximate. If activities are work related in the same condition, it is set as W_Proximate. The variables used for this BN and their possible states are summarized in Table 6. This Bayesian network is modeled to infer two types of relationships: private and work relationships. Inferred result is used to make mobile social networks by social network manager in the next step.
Variables and the possible states of Bayesian network model designed.
3.4. Social Network Manager
A social network manager constructs a mobile social network using mobile logs and inferred high-level contexts. Generally, a relationship between users in a mobile social network considers only the existence (or strength) of connection. This paper identifies semantic relationships between users and builds the mobile social network with them. These relationships are inferred from the Bayesian network in Figure 4.
The mobile social network 𝒩 is defined as 
We also need to decide how mobile social networks can be displayed. This network is represented as a graph where the nodes correspond to mobile users and the links correspond to relationships between them. The type of relationship mentioned above decides its thickness, and the importance of users decides their size in a graph. To calculate this importance, the closeness centrality is used as follows [21]:
3.5. Service Manager for Context Sharing
In a mobile client, a service manager provides the context sharing service to users based on the mobile social network made by the social network manager. Context sharing service includes sharing information of friends registered in user's phonebook. Also, it gets map images of locations that GPS coordinates indicate from Google Maps (http://maps.google.com/). A contact list in a phonebook can be a mobile social network centering users themselves potentially. The proposed service provides the contact list with user name, image, activity, and emotion if users in that list are close enough. That is, private information like activity or emotion is shared between close friends. An application of this context sharing service will be discussed in the experiments later.
4. Experiments
4.1. Environment and Data
The proposed system has a server written in C# (.NetFramework3.5) based on Windows platform and mobile clients that contain a mobile log collector and ContextViewer written in C# (.NetCompactFramework3.5) using SAMSUNG T*Omnia SCH-M490, a smart phone based on Windows mobile 6.5. MySQL 5.1 is used as a database management system. The server and the mobile clients communicate with each other through TCP/IP socket.
In order to model and evaluate Bayesian networks appropriately, we collected mobile logs from eleven graduate students for three weeks. Participants consist of one female student and ten male students, and eight students have experiences of using smart phones before while the others have not. We asked them to annotate high-level context information of activity and emotion manually. Three weeks seem to be too short, but Eagle et al. analyzed that long-term data from mobile devices have enormous redundancy and observation for two weeks can largely replicate the data produced from nine months observations [22].
Tables 7 and 8 show parts of mobile logs collected by a user (User 1). Table 7 provides a phone call log, which includes sender, start time, end time, status, and receiver. The last four digits of cell phone numbers are replaced by “xxxx” for the sake of privacy. Table 8 shows an activity log of the same user. Activity and emotion have been labeled together with start and end time.
A part of phone-call log of User 1.
A part of an activity log of User 1.
4.2. Evaluation of Bayesian Networks
In order to evaluate Bayesian networks, inference models in the system, we calculate the recall and precision of Bayesian networks for activity inference by using collected mobile logs of 11 individuals. Precision can be seen as a measure of exactness, whereas recall is a measure of completeness. The result is shown in Figure 5. The probability is 80~95% about “study,” “meal,” and “sleep.” However, “play” and “rest” are low because these activities are done irregularly and regardless of environment. It is also unusual that an activity “meal” has a low recall rate while it has a high precision. It is because “meal” happens both indoor and outdoor. Outdoor “meal” can be recognized easily, but indoor “meal” is confused with other indoor activities like rest and study.
A result of evaluating Bayesian networks for activities.
The inferred relationships are compared with actual relationships from self-reported data. Users were asked to select one of the predefined relationships to other users. Options are “close friend/friend/acquaintance/none” for private relationship and “close colleague/colleague/acquaintance/none” for work relationship. Answers of two users are not always the same, and in those cases, we decided that the inference was correct if the inferred answer was the same as either of two different answers. Table 9 provides accuracy, precision, and recall of this comparison.
Accuracy in comparison of actual and inferred relationships.
4.3. Mobile Context Sharing Application
Using the proximity and inferred high-level contexts together, we can make various mobile social networks. For example, two users are close in private relation if they often watch movie together. Here, we present two networks as examples.
A network based on private relationships: First, we built a mobile social network based on private relationships between users, as shown in Figure 6. Socializing activities like meal and play or active leisure activities like sport took important roles in this network. In this network, the size of a node represents the importance of a corresponding user, and the position is based on their physical location. In reality users 3, 4, 7, and 10 share an office, and the other users share the other office. Thicknesses of links are decided by the relations inferred. A network based on work relationships: Figure 7 shows a mobile social network built based on work relationships. Obviously, activities related to work (study) are significant in this network. We found two large groups in the network (solid lines and dotted lines), and it is found that users in the same group work at the same office. In short, this network depends on the proximity very much. The network is visualized with the same method as private relationship-based network, and user 1 is shown as the most important person in this network. In the real world, user 1 is the manager of the lab and a main room. In terms of the relationships, this network depends on the proximity very much.
A mobile social network built based on private relations between users.
A mobile social network built based on work relations among users.
Figure 8 shows an example of using the system. When the user executes ContextViewer, an application in the mobile client, a phonebook is displayed (Figure 8(a)). In the interface, the user sees only friends who allow the user to view their contexts and the user does for them. The user catches that “Yoon” feels upset. The user wants to talk with him and encourage him. When the user selects his item, a map browser is opened (Figure 8(b)). The user knows where he is and if he is near the user. The user can call him and meet him.
ContextViewer.
4.4. The Usability Test
In order to validate the usefulness of the proposed system, we performed a subjective test about the implemented application for thirteen subjectives based on the System Usability Scale (SUS) questionnaires. The SUS is a simple, ten-item scale giving a global view of subjective assessments of usability. Table 10 shows ten items for questionnaire, and each subjective should answer the item as 5-degree scale where one means strongly disagree whereas five means strongly agree. Total score from ten answers has a range of zero to one hundred for each subject [23]. It can be thought that the score of 50 is neutral, and the system with more than 50 has a good usability. User 6, who gave the lowest SUS score, has proved that he has never used a smartphone. Figure 9 shows the SUS test results of thirteen participants, which indicate that the system provides an effective way for sharing contexts conveniently. By questionnaire, numbers 3 and 5 have been given the best score while number 9 has been given the worst score. Questionnaires with the best score include very important parts (no. 3 easy to use and no. 5 well integrated functions) in evaluating mobile services.
The available values for activity and emotion.
SUS scores for the proposed system.
5. Concluding Remarks
In this paper, we have presented a context sharing system in mobile environment that recognizes and shares high-level context information automatically to support high-confidence cyber-physical systems. In the proposed system, a mobile client collected mobile logs, sent them to a server, and provided context sharing interface. ContexViewer is a service to share contextual information of friends. It consists of a phonebook and a map browser. The server preprocesses and recognizes high-level contexts with Bayesian network models to handle uncertainty in mobile environment. When the mobile client requests some contexts, if not permitted, the server prevents to send them. The usefulness of the proposed system is shown by evaluating Bayesian network models that infer activity and user relationship and by performing SUS test of our context sharing application. The proposed system helps users to share context information without manual setting.
There are several ongoing tasks in this work. Bayesian network models designed in this paper are for general users, and it cannot work well for all substantial users. A few more models for different types of users will be helpful to make this model more general. It is also required to compare the proposed context sharing system with the alternatives. Thirdly, we are planning to apply the proposed method to other applications. The ContextViewer only shows the context information, but it can provide more powerful services based on high-level contexts. For example, behavior recommendation for mobile user can be possible based on diverse contexts. Finally, we are using the context information collected from smart phones, but the source of context information can be extended to a smart space, which has diverse sensors in an environment for intelligent services.
Footnotes
Acknowledgment
This research was supported by the Original Technology Research Program for Brain Science through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2010-0018948).