A novel context-aware augmented reality framework for maintenance systems

Abstract

Augmented Reality (AR) bridges the gap between the real and the virtual world by bringing virtual information to a real environment as seamlessly as possible. The need for better perception of knowledge-intensive complex maintenance tasks and access to large amounts of documents and data makes the use of AR technology promising in a maintenance domain. Context-awareness enhances the usability of such AR applications, i.e. the output and behavior of the system will be adapted according to different contexts, such as the user location, preferences, devices, etc. to afford a higher level of personalization. The adaptation needs to be efficient in terms of performance and speed. This paper presents an optimized framework which combines context-awareness and AR for training and assisting technicians in maintaining equipment in an industrial context to improve field workers effectiveness. Ontology is used to model a maintenance context, and Semantic Web Rule Language (SWRL) provides logical reasoning. This optimized framework utilizes a behavior network to select a collection of suitable actions based on the current step of an ongoing task, and applies context-based inferred information from the ontology to each member of this collection. Evaluation results comparing the performance of the proposed framework with conventional ontology alone in a maintenance domain confirmed that the proposed framework in this research provides the same results as the ontology in terms of content, but it runs much faster in terms of run-time and performance. The proposed context-aware framework is quite valuable especially in terms of response time and performance for maintenance systems with a large number of maintenance activities.

Keywords

Context-awareness augmented reality behavior network ontology operation and maintenance

1. Introduction

Efficient Operation and Maintenance (O&M) guarantees reliability, safety, and energy saving of equipment. O&M includes actions required to upkeep equipment and facilities and helps to lengthen the equipment lifecycle [14,21]. The complex structure of equipment means that technicians face difficulties to conduct maintenance activities [26] and that access to technical paper-based manuals is required especially when these activities are performed infrequently [18].

Computing technologies have enabled developers to support maintenance activities faster, more reliable and more intuitive. As a consequence, traditional paper-based manuals are on the verge of extinction [8]. AR technology, which has been used in various domains including maintenance [22], is technology which can be employed to assist users completing steps of a maintenance task by overlaying virtual objects on a real-time equipment scene and providing human-machine interaction as well [4,22]. AR technology improves a user’s perception of maintenance activities by superimposing multimedia alerts and providing bi-directional visual interaction between the user and the system [14]. However, context-awareness, which is an integral part of improving systems usability, needs to be employed in order to adapt the system’s behavior and output based on a user’s context [26].

Context is defined as a set of states and settings in the environment that determines an application’s behavior or in which an application event occurs that is of interest to the user [3]. Context can be divided into three groups, namely, computing context (network bandwidth, communication cost, etc.), user context (user profile, people nearby, etc.) and physical context (light, noise level, etc.) [16]. The following examples of context will make it clear. A user location can be used to alert on future preventive jobs about equipment nearby to the user. As another example, a user’s expertise level, defined as user context, affects details of the information to be shown. A user with a high expertise level needs less details to perform a task, but a novice user requires more details to help him finish the job. Suppose a user which runs the system on his tablet device. If the battery level of his device goes below a certain value (e.g. 25%) or the CPU usage of his device goes higher than a threshold (e.g. 85%), then the system should avoid overlaying 3D models that consume more battery and need more processing resources. A textual overlay should be used instead.

In addition, one should keep in mind that the goal of providing context-awareness within AR-based maintenance systems is to improve efficiency and reduce the time, cost and effort required to complete maintenance tasks. However, the achievement of this goal should not affect the performance of the proposed systems in case of run-time and resources. The aforementioned systems need to be able to run as fast as possible even on devices with limited computational resources, and they should not take an intolerable amount of time to provide output for the final user.

This paper presents an optimized context-aware AR-based framework to assist technicians conducting maintenance tasks. The proposed framework makes use of collected raw context data and performs reasoning on them to obtain new inferred data and to provide adaptive behavior using an ontology. On the other hand, the responsibility of selecting the collection of information instances to assist the technician for each step of ongoing maintenance routine is granted to a behavior network. The latter inconsiderably improves the performance of the system instead of running time-consuming SWRL rules.

The rest of this paper is organized as follows. Section 2 gives a review of context-aware systems in various domains. Section 3 shows the architecture of the proposed framework in this paper. An experiment of the proposed framework is presented in Section 4. Results of employing the proposed framework are provided in Section 5. Section 6 summarizes the paper and introduces future works.

2. Related work

Conducting activities, such as scheduling, periodical controls, performing preventive and predictive procedures, ensures that property and equipment achieve their intended goals. Preventing breakdowns, increasing efficiency, reliability and safety are the main purposes of operation and maintenance activities [8,21].

AR technology can be utilized to train complicated procedures and provide maintenance instructions as well as visual interactions to onsite technicians and it has proved to be useful in a maintenance context [26].

Wang et al. developed a framework for an AR-based system to conduct complicated maintenance tasks cooperatively and efficiently through a local network infrastructure between technicians and remote experts [23]. Schwald et al. proposed an AR system to aid technicians in maintaining equipment in an operational condition as well as instructing them how to use facilities [18]. AROMA-FF encompasses an AR-based interface in order to superimpose 3D assistive models and a maintenance information model including equipment settings, sub-systems, components, actors, maintenance history, sensor-based live data stream and geometric representations [8].

To reach the goal of improving the usability of AR maintenance systems, there is a need to utilize context-aware computing [25] in order to discover contextual information and use them to provide more practical information to the users [3]. As a result, virtual information will be changed according to various context [26] and the intelligent behavior in a smart environment will be improved [15].

Gouin-Vallerand et al. employ different types of contextual information including user profiles, device profiles, etc. and propose a context-aware service provision to be used in smart environments [5]. The CoMotion project at MIT exploits GPS-based user location as a context parameter and plays back a recorded message automatically as soon as a recipient arrives at the desired location [11]. CyberGuide, an information provider service, takes advantage of both user location and orientation to provide information such as directions, background information, places of interest and people comments for the tourists [1,9]. A notepad application is developed by Schmidt et al. for a PDA device to adapt the font size and backlight based on the user’s context, such as moving state and light level. In another scenario, a mobile phone profile is changed depending on the current phone state [17]. Oh et al. suggest CAMAR, which is a mixture of mobile AR and context-awareness. CAMAR makes use of unified user context information and allows users to view and tailor augmented information to share them with other end users [13]. Mileo et al. present a context model that makes use of information from pervasive sensors placed in the environment and patient explanatory feedback as well to interpret complex collected data and reason about available knowledge using logic programming language in order to support patient well-being [12]. Byun et al. employed context history beside user modeling and machine learning algorithms to provide adaptation. The proposed system collects and learns user preferences about the office window state in different room conditions. Afterwards, the user desired window state will be induced based on the gathered data, i.e. user behavior [2].

Context-awareness is not widely used due to the lack of effective infrastructure to support context-aware applications [15]. Ontology has been widely used for context modeling to define concepts in a specific domain and relations between concepts [19]. Ontology provides benefits such as knowledge sharing, logic-based reasoning and inference, and knowledge reuse [24]. Entities and their relations are introduced in the form of classes, subclasses and properties in ontologies. Each class can have several subclasses, individuals and properties. Properties are divided into two types, object and data properties. Object properties define a relation from an individual to another individual. Data properties define a relation from an individual to a data value such as a String, Boolean, Date, Time, etc. [7].

A formal general reusable ontology-based context model, including a default vocabulary and extendable structure, is provided by Hoque et al. to support ontology-based reasoning and rule-based reasoning in a smart home environment [6]. CONON takes care of context modeling and ontology reasoning in addition to user defined reasoning in a pervasive computing environment. It defines a basic context ontology consist of general concepts and the ability to extend ontology for a specific domain hierarchically [24]. A combination of first order probabilistic logic and web ontology language (OWL) is implemented as an agent-based middleware by Qin et al. to gain contextual information, knowledge demonstration, usage and sharing in context-aware applications [15].

Ong et al. present a novel ontology-based context-aware AR maintenance system to support the provision of context related information to the technicians. In addition, the system provides a real-time collaborative interface for technicians to ask for help from remote experts or vendors as well [14]. Another context-aware maintenance application based on ontology equipped with AR technology with the ability to run on a tablet PC is introduced by Flatt et al. This software superimposes maintenance information beside live operational data as virtual information on screen and allows users to attach sticky notes to the equipment components enriched with position information and makes them recognizable later even in different camera positions [4]. Zhu et al. introduced an Authorable Context-aware Augmented Reality System (ACARS) using ontology as a context modeler to support technicians in maintenance jobs and enable them to interact with AR contents. As a result, they can create new augmented information themselves or revise formerly created information [26]. Authoring for Context-Aware AR (ACAAR) is another authoring tool to create augmented context-aware information in maintenance AR applications with little programming skills by employing ontology. ACAAR allows the technicians to correlate virtual information with different contexts [25].

Behavior Network takes responsibility for the problem of action selection in an agent-based environment. Each agent faces a collection of competence modules. Changes in the activation level of each module will lead to its activation. The agent will select the modules with the highest activation level greater than desired threshold [10].

Song and Cho present a context-adaptive user interface for the simplification of using different control devices containing different functions on each device in a home environment. The proposed system takes advantage of a Bayesian network to predict the desired devices in the user current situation and then employs a behavior network to select the required functions at each step hierarchically [20].

The literature review shows that many context-aware systems have been designed and implemented in various domains. However, context-awareness has been disregarded in the maintenance domain and there is not much research done in this field. Thus, most of the maintenance AR applications have been deprived of promising benefits of context-awareness, such as improved usability. But while improving usability due to context-awareness, performance of the system needs to be considered. Conventional ontology-based context-aware maintenance systems are in charge of defining what needs be shown in each step of a maintenance task using SWRL rules. These rules are time-consuming and as they grow the performance of the system decreases. In an industrial environment with 10,000 facilities, where each facility has more or less maintenance tasks with 10 steps or more to complete the job, applying rules and inferring the relevant output of each step by using a reasoner will be time-consuming and it is not bearable for the user to wait minutes for the system to start working.

In this paper, we propose a context-aware framework as a base for an AR maintenance application. The proposed system employs an OWL ontology for context-awareness, and uses behavior network to select actions to be conducted at each step as a supplement to improve run-time speed. Thus, the ability of running applications on devices with limited processing resources will be provided and the system starts working in a reasonably acceptable time.

3. System overview

This section describes the process of development of a context-aware AR maintenance system. As stated before, AR can superimpose maintenance information instances on real equipment and describe a virtual equivalent of a specific type of information, such as paper-based manuals, usage instructions, repair procedures, etc. to the users.

3.1. Classification of information instances

Information instances are those virtual objects to be superimposed on the environment using AR. To provide information instances for maintenance systems, two questions need to be taken into account. The first one is the time when a procedure should be performed, and the second one is how a procedure should be carried out. In this research, information instances are categorized in three groups based on maintenance activity types which will give the answers to the two questions stated above.

3.1.1. Static information

In break-down activities, a specific routine consists of definite steps to be performed on an equipment that has stopped working due to a failure reason in order to return that equipment back into a normal functional condition. Operational activity, however, is not a maintenance activity but is included in this research as a maintenance routine containing definite steps advised by the manufacturer on how to run an equipment correctly. These types of information are named as static information considering the fact that they do not vary a lot during the equipment lifecycle.

3.1.2. Periodical information

Technicians need to perform some procedures on equipment based on the manufacturer schedule in order to avoid failure and further break-down costs. This schedule introduces the conditions to perform a job or to send an alert to the technicians. These conditions include predefined intervals, such as every eight operating hour, every day, every thousand kilometers traveled, etc. or certain parameters, such as pressure, current or noise level higher than a certain threshold. When one of the conditions stated above is met, some specific steps should be executed on that equipment. Since information included in this category is based on a schedule, it is called periodical information.

3.1.3. Dynamic information

Monitoring activities make engineers able to control live operational data as well as statistical reports. As this information changes continuously, this category is named as dynamic information.

3.2. AR-based maintenance system

Different types of information instances described in 3.1 can be presented using AR technology. For this to happen there should exist three phases in each AR system, as shown in Fig. 1, namely Identification (P1), Retrieval (P2) and Visualization (P3).

Fig. 1.

Phases of AR systems.

Identification (P1) is responsible for recognizing equipment in the environment automatically by means of an equipment’s natural features or visual markers attached to the equipment by employing computer vision-based object recognition algorithms, RFID devices, GPS-based location and orientation, etc. or manual input of a user into the system specifying the equipment. The proposed system in this research uses natural features to detect and track equipment in the environment. To do so, the system needs to be trained in advance. Therefore, an image of the equipment should be fed to the system for the sake of feature extraction. Finishing this procedure will create a dictionary containing natural features of the equipment, which can be used for further detection and tracking purposes. Figure 2 shows an original image taken from an equipment and Fig. 3 shows the extracted features of this equipment.

Fig. 2.

Original equipment image.

Fig. 3.

Equipment image feature set.

The second phase in AR-based systems is Retrieval phase (P2) which is in charge of obtaining maintenance related information or live data through various databases and pass them to the next phase.

Visualization (P3), the last phase, integrates retrieved information and superimposes them on real equipment scenes obtained through live video streams. AR technology fulfills the requirements to implement these phases.

3.3. Adaptation scheme

The important fact is that showing static virtual information to the user is not satisfactory. So output information needs to be selected based on a user’s context such as expertise level, skill name, interested color, location, etc. in Retrieval phase (P2) which context-awareness seeks to fulfill in AR-based maintenance systems. Therefore an adaptation engine is considered within Retrieval phase (P2) as shown in Fig. 4. This scheme is the basis for the provision of relevant information to the users in the terms of content and presentation.

Fig. 4.

Adaptation engine.

The Adaptation Engine as shown in Fig. 4 is the heart of Retrieval phase (P2) and is divided into two subsections referred to as the Selector and Renderer. The Selector receives real time context, predefined logical rules and instances from ontology as input. After applying logical rules on the received context, the Selector chooses which combination of virtual information needs to be retrieved from the maintenance database to be displayed on the scene using a behavior network. This combination is next passed to the Renderer which applies rendering properties such as color, transparency, etc. considering the rules defined in the ontology by system developers. Output generated by this module will be based on what needs to be displayed on the screen.

3.4. System architecture

The proposed system consists of five different modules, namely, the System Core, Context Collector, Behavior Network, Reasoner, Visualization and databases to provide context-aware information instances to the final user, as shown in Fig. 5. The System Core module is called initially whenever the system starts working, and it establishes the links between different modules. The Context Collector is responsible for providing real time context to the System Core. The Behavior Network prepares a combination of instructions to be displayed from the maintenance database based on the rules step, individuals and properties defined in the system ontology and return them back to the System Core module. Applying the rules defined in ontology over individuals and sending back the results to the Behavior Network is the Reasoner’s duty. The Visualization module receives the combination of output information from the System Core and presents them on the screen. Embedded modules interact with each other as well as some other external libraries. A description of each module is provided as follows.

Fig. 5.

System architecture.

3.4.1. System core module

As Fig. 5 illustrates, the System Core module is the start point of the system and will be called initially whenever the system starts. For the system to function, there is a need for a controller to provide a link between the different modules. This controller triggers other modules whenever needed, and retrieves results after process completion in the fired module. The System Core plays the role of a controller in the proposed system.

First and foremost, the System Core calls for the Context Collector to obtain real time context. After collection of these raw context data, they are passed to the Behavior Network module via the System Core to choose output information accordingly. Acquired combination of information will be passed to the Visualization module in order to be displayed on the screen.

3.4.2. Context collector module

This module has a main class referred to as Context Provider which is called by System Core module. This class asks for real time context which can be obtained in two methods.

The first method is by reading live sensor data, such as the time of day, current date, battery level, CPU usage level and etc. automatically. The second method is gathering information through manual user input such as person expertise level, person preferred color and etc. The Context Provider class transmits these raw context collected data to the System Core for further processing.

3.4.3. Reasoner module

In order to describe the maintenance environment, context modeling is needed which defines entities and relations between them. Context modeling plays an important role in the process of implementing context-aware applications in order to provide information according to the status of people, equipment, activity and etc. Following section provides a description of the context modeling used in our proposed system.

Fig. 6.

CAMON ontology.

Ontology This research adopts the CONON ontology proposed by Wang et al. as the base ontology [24]. Some customization and extension are being done upon it to model a maintenance context in order to provide a common definition for certain maintenance concepts and their inter-relations. The proposed ontology in this research is named CAMON as an abbreviation for Context Aware Maintenance ONtology. As illustrated in Fig. 6, CAMON consists of five base classes named Activity, CompEntity, Information-Instance, Location and Person. Activity subclasses represent different maintenance activities described in Section 3.1. CompEntity includes CompDevice, Equipment and SensorDevice as subclasses. CompDevice represents a user’s computational devices, such as mobile, tablet, etc. which are used to run the system during a maintenance activity. Equipment represents facilities that the technician will perform an activity on. SensorDevice holds AR markers placed in the environment. Location is divided to IndoorSpace and OutdoorSpace. Person represents everybody that is involved in a maintenance activity. BDM_Engineer are those engineers who carry out a corrective maintenance activity. PM_Engineer represents those who perform a preventive activity routine on an equipment. Manager is a delegate of persons who are involved in monitoring activities and Operator represents persons who have decided to run an operation on an equipment. InformationInstance holds the related rule name of each activity to be displayed over the scene.

Based on the maintenance environment, these classes can be changed or extended and individuals may be inserted into the ontology. For example, the definition of ChangeFilter as an individual of class Activity is illustrated in Fig. 7. This activity is conducted on an equipment named CarEngine, uses information named ChangeFilter_Rule as information instance, failure reason is stated as DirtyAirFilter and current status of activity is at Step 1.

As another example, an individual named John is created as an individual of Person class as illustrated in Fig. 8. John is engaged in a ChangeFilter activity. He is in charge of an iPhone6 device. His expertise level is defined as Low, the preferred color is saved as Red, and the skill name is stated as Mechanic.

Fig. 7.

Create an individual of Activity class.

Fig. 8.

Define an individual of Person class.

Logical rule definition Logical rules which describe the environment in detail will be defined using SWRL rules. Any SWRL rule consists of two parts named Body and Head. Body contains a set of preconditions and Head contains a set of actions which will be taken into account whenever the Body preconditions are satisfied completely. An example of SWRL rule is presented in Fig. 9. In this example, if atoms A(?x) and B(?x) are satisfied, action C(?x) will be taken and the result will be true. A, B and C can be classes, object properties or data properties and x introduces a variable in this example. SWRL built-in functions such as swrlb:equal can also be used in SWRL rules.

Fig. 9.

An example of SWRL rule.

Fig. 10.

An example of environment rule.

Fig. 11.

An example of context adaptation rule.

SWRL rules provide the ability for adaptation and context reasoning, i.e. inferring high-level context from raw context data. In this research logical rules are divided into three types:

Environment Rules: This type of rule defines complex relations between entities, and introduces the environment in more detail. An example of this type of rule is illustrated in Fig. 10. System developers will declare these types of rules based on the environment in which the system will run.

Context Adaptation Rules: These rules provide the ability to deduce high level context from raw context data. An example of this type of rule is shown in Fig. 11. This example uses real time user’s device CPU usage, and says that if the CPU usage of the user’s device is greater than 80% then the related information instance presentation type needs to be text-based in order to avoid CPU overload as a textual presentation consumes less computing resources.

Different types of raw context data can be used to infer more usable information and provide a robust adaptation.

Static Adaptation Rules: The rules that introduce the requirements for a specific maintenance activity to be fired, will be described here. Actually these rules are simple ones which say which rule should be considered in which situation. But SWRL rules regarding what should be displayed in each step of the selected routine are omitted in our proposed system and Behavior Network is responsible for this job. An example of static adaptation rule is illustrated in Fig. 12.

The rule shown in Fig. 12 says that if there is an activity for which the failure reason is DirtyAirFilter, then the related information instance will be ChangeFilter_Rule.

Fig. 12.

An example of static adaptation rule.

Reasoner module Reasoner module performs two types of reasoning. The first one is Ontology Reasoning. Using the Ontology Reasoning method, the reasoner classifies individuals based on their definitions and assumes each individual as an individual of a class. As an example, in the ontology it is stated that if an activity has a failure reason it is assumed to be a corrective activity therefore if Activity1 belongs to Activity class and it has a data property of FailureReason set to some certain value then the Reasoner infers that this activity is a corrective activity.

The second reasoning method receives raw context data from Behavior Network module, applies predefined SWRL rules on the ontology, and transmits the result information instances be displayed with data properties such as status, level of detail, text color, transparency, etc. to the behavior network.

3.4.4. Behavior network module

Once we determined the output information instance name such as Activity1_Rule with its relevant properties, a behavior network will be generated using activity set ${S}$ as sensors like Eq. (1), activity set ${G}$ as goals like Eq. (2) and activity instructions as middle nodes ${N}$ like Eq. (3). Each activity holds corresponding properties of InformationInstance class stated in the proposed CAMON ontology. $\begin{array}{l} (1) & S = {{activity}_{1}, {activity}_{2}, {activity}_{3}, \dots} \\ (2) & G = {{activity}_{1}_done, {activity}_{2}_done, \dots} \\ (3) & N = {{node}_{1}, {node}_{2}, {node}_{3}, \dots} \end{array}$

Each middle node has a structure like Eq. (4). $c_{i}$ shows a list of preconditions for ${node}_{i}$ to be activated, $a_{i}$ is the node add list and $d_{i}$ holds its delete list. $\begin{matrix} (4) & {node}_{i} = {c_{i}, a_{i}, d_{i}, \propto_{i}} \end{matrix}$

A specific collection of nodes shows different steps of a maintenance procedure. If an activity contains seven steps for completion, seven distinct stages containing middle nodes will be considered in the network for that activity. Based on the value of LevelOfDetail property each stage can have one or more parallel middle nodes. Figure 13 illustrates an example of middle nodes in a maintenance task with three steps in a behavior network when the value of LevelOfDetail is inferred as high demanding more detailed instructions.

Fig. 13.

Middle nodes in a behavior network.

After generating middle nodes, successor and predecessor links will be generated for each middle node. There exists a successor link from node x to node y if node y is in the add list of node x and node x is a precondition of node y, i.e. for every proposition $p \in a_{x} \cap c_{y}$ there is a successor link from node x to node y. A predecessor link will be created from node x to node y when there is a successor link from node y to x or for every proposition $p \in c_{x} \cap a_{y}$ . Each node has an activation level $α_{i}$ . When a node activation level surpasses a threshold that node will be activated which causes the node source code to be executed.

This will cause the generation of a logical path from each member of sensor set ${S}$ to the relevant member of goal set ${G}$ . The concept is to start the process from sensors and spread an energy through links to activate the desired middle nodes in the network en-route selected maintenance task specific steps. Behavior network in this research is implemented using a graph which is surfed using Best First Search (BFS) method up to the time when module arrives to the ongoing activity step.

As shown in Fig. 14 considering that Task 1 sensor with high level of detail is the desired sensor and the technician is conducting this procedure at step 2 therefore the relevant goal is Task1_done and the behavior network returns stage 2 middle nodes including Instruction2 (Description1) and Instruction2 (Description2) with their predefined properties. First the add list of the selected sensor which are stage 1 middle nodes will be activated and they activate middle nodes of stage 2 which are intended nodes.

At last a collection of middle nodes will be sent to the Visualization method. The structure of such output collection is shown in Fig. 15. Each member of this collection includes three properties, namely, item data, text color and presentation type. Item data represents the content of the output. Text color and presentation type are display properties.

Fig. 14.

Process of retrieving instruction steps.

3.4.5. Visualization module

The Behavior Network module prepares and sends a set of output objects like the one showed in Fig. 15 to the System Core module. The System Core module receives this set and transmit them to the Visualization module. The input video stream will be obtained using the user’s device camera in the Visualization module and members of the received output information list will be augmented on this video stream. Then the processed video stream including augmented information will be displayed on the user device display screen. Therefore the final user can make use of the maintenance scene enriched with virtual added guidance in order to complete the intended instruction.

Fig. 15.

Transmitted pack.

3.5. Implementation

This section provides the details about implementation of the proposed architecture in this research. The System Core, Context Collector, Behavior Network and Reasoner modules are developed using Java Development Kit (JDK) 1.8. IntelliJ Idea is used as an IDE for developing source codes. Microsoft SQL Server 2005 serves as the database server. The Behavior Network module connects to a SQL Server using the jdbc-4.0 external library in order to obtain information instances and create the required behavior network. The Reasoner module uses owlapi-osgidistribution-4.1.0 external library for reading ontology classes, properties and individuals. The Pellet 2.2.0 library is used by the Reasoner module for further Ontology and SWRL reasoning. The Visualization module is implemented using the $C + +$ programming language and uses ARToolkit to implement AR and superimpose virtual objects over the environment.

4. Experiment

The proposed framework provides assistance to the users to perform maintenance tasks based on AR technology. An experiment has been conducted in a simulated maintenance environment to validate the output results of the proposed framework. In the simulated environment, the user tries to change the air filter of a car engine in different contexts. The results of the experiment are as follows.

In the first experiment, a novice BDM engineer is changing the air filter of a car engine. The engineer’s preferred color is set as red and his device CPU usage is obtained as 75 percent. As Fig. 16 illustrates, a red box and a guidance string introducing the location of the air filter container is shown to the user with a user preferred color and a symbolic presentation due to the high CPU usage.

The engineer continues to the next step of the task in the second experiment. The difference with the previous step is that CPU usage drops to 30% and the battery level shows 80%. Therefore, the intuitive presentation is selected to help the user opening the screws and circled arrows will be shown over the screws location to show the user that the screws need to be opened anticlockwise at this step (Fig. 17). A caution sign is also presented on the scene as an alert to the novice user of the system.

Fig. 16.

Change air filter – step 1.

Fig. 17.

Change air filter – step 2.

In the third experiment, the user device battery level drops to 25% in the second step which caused the presentation type to be changed to symbolic, and also the user preferred color changed to green. So, the arrows and caution sign are deleted and the simple green circles and a text are presented on the scene instead, as shown on Fig. 18.

The forth experiment starts when the user surfs to the third step of the task. Considering the user preferred color as blue, CPU load at 25%, and the battery connected to AC power, the intuitive presentation is selected as the presentation type. Therefore, a blue 3D arrow with related guidance to open clamps is presented on the scene (see Fig. 19).

Fig. 18.

Change air filter – step 2.

Fig. 19.

Change air filter – step 3.

5. Results

A general maintenance environment is being modeled in this research. We implemented a middle-ware to produce sample data to evaluate the proposed framework. Using this middle-ware the same sample data is produced for two cases: One for the time when the ontology is used alone, and the other for the time when the proposed framework in this paper uses a mixture of ontology and behavior network.

Sample data in each ontology is divided into two categories: First, when an expert technician is running the system resulting in a low level of detail. Second, the time that a technician with low expertise level is using the system which results in high level of detail output information.

For each of these cases the relevant ontology is filled with the desired number of individuals as Table 1 in order to evaluate the performance of the proposed framework.

Table 1
Number of individuals

Class Name No. of individuals

Person 1

Device 1

Location 18

Equipment 20

Activity 1, 5, 10, 15, 25, 35, 50, 75, 100, 125, 150

Class Name	No. of individuals
Person	1
Device	1
Location	18
Equipment	20
Activity	1, 5, 10, 15, 25, 35, 50, 75, 100, 125, 150

A performance evaluation is conducted in order to compare the system run-time using each method. For first time the comparison is made between ontology alone and the proposed framework in this paper considering user expertise level as high. Run-time to achieve the output information instances and their related properties in this case is shown as Table 2.

Table 2

Run-time comparison when LoD is low

No. of Activity	Ontology Alone (seconds)	Proposed Framework (seconds)
1	2.334	3.479
5	3.255	3.695
10	3.402	3.766
15	3.608	3.847
25	4.427	4.048
35	5.457	4.060
50	7.282	4.313
75	7.517	4.364
100	9.246	4.496
125	15.126	4.648
150	17.019	7.638

Fig. 20.

Run-time comparison when LoD is low.

As shown in Table 2 and Fig. 20, in the beginning where the number of activities is small, the run-time of ontology alone method is better than the proposed framework. But as the number of activities grows, the application run-time using ontology increases exponentially. On the other hand, the run-time of the proposed framework in this paper increases slightly and the average reduction rate of 2.756 seconds in run-time is obtained when using this method.

This evaluation is conducted in another case when the user’s expertise level is low, which leads the activity level of detail to be high and causes greater number of information instances to be retrieved. The run-time for this case is shown in Table 3.

Table 3

Run-time comparison when LoD is high

No. of Activity	Ontology Alone (seconds)	Proposed Framework (seconds)
1	2.438	3.437
5	3.180	3.636
10	3.660	3.737
15	3.721	3.827
25	5.470	3.965
35	5.749	4.197
50	7.296	4.535
75	8.097	4.547
100	9.916	4.601
125	20.264	4.766
150	21.886	8.324

As shown in Table 3 and Fig. 21, the proposed framework achieves a better score in both cases. The average reduction rate in run-time considering this case is 3.828 seconds.

Fig. 21.

Run-time comparison when LoD is high.

We achieved a total average reduction rate of 42% in the initial run-time when the proposed framework in this paper is used which is more satisfactory to the end users.

6. Conclusion and future work

In this research a review of simple AR-based maintenance systems, context-aware systems and AR-based context-aware maintenance systems has been conducted. A context-aware framework for AR systems in maintenance domain is proposed. The proposed framework employs ontology and behavior network.

Ontology models the maintenance context and defines entities in the real environment and their inter-relations. Applying logical SWRL rules and reasoning based on the current user’s context in order to infer and provide context adaptive output to the final user is also the responsibility of ontology and the implemented reasoning engine.

The need for defining SWRL rules to select output information collection according to the ongoing activity step is ignored in the ontology, and turned over to a behavior network instead which selects the collection of output information to be superimposed on the screen based on the current activity step. This will reduce the number of SWRL rules defined in the ontology and decrease the time required to run the framework.

There are things in this research to be improved as future work. First, although context-awareness helps to improve the usability of AR systems, there is a lack of research in this field in maintenance context and there are different context parameters that can be employed to provide a better adaptive user interface to the users. Second, the lack of an authoring tool is sensed in this research in order to provide a bi-directional interaction tool between technicians and the system to provide maintenance procedure steps by technicians and make corrections on data provided by developers. Third, with the help of context-awareness, a suggested list of technicians with the same skill and location is provided, but there is a need for a tool to provide remote collaborative communication between technicians and the members of this list. Finally, there is a need to evaluate the proposed adaptive user interface in terms of usability, learnability, time required for job completion, etc.

References

G.D.

Abowd,

C.G.

Atkeson,

Hong,

Long,

Kooper and

Pinkerton, Cyberguide: A mobile context-aware tour guide, Wireless Networks3(5) (1997), 421–433. doi:10.1023/A:1019194325861.

H.E.

Byun and

Cheverst, Utilizing context history to provide dynamic adaptations, Applied Artificial Intelligence18(6) (2004), 533–548. doi:10.1080/08839510490462894.

Chen and

Kotz, A survey of context-aware mobile computing research, Technical Report TR2000-381, Department of Computer Science, Dartmouth College, 2000.

Flatt,

Koch,

Röcker,

Günter and

Jasperneite, A context-aware assistance system for maintenance applications in smart factories based on augmented reality and indoor localization, in: 20th IEEE International Conference on Emerging Technologies and Factory Automation, IEEE, Luxembourg, 2015, pp. 1–4.

Gouin-Vallerand,

Abdulrazak,

Giroux and

A.K.

Dey, A context-aware service provision system for smart environments based on the user interaction modalities, Journal of Ambient Intelligence and Smart Environments5(1) (2013), 47–64.

M.R.

Hoque,

M.H.

Kabir,

Thapa and

S.-H.

Yang, Ontology-based context modeling to facilitate reasoning in a context-aware system: A case study for the smart home, International Journal of Smart Home9(9) (2015), 151–156. doi:10.14257/ijsh.2015.9.9.16.

Horridge,

Jupp,

Moulton,

Rector,

Stevens and

Wroe, A practical guide to building OWL ontologies using Protégé 4 and CO-ODE Tools Edition 1.2, The University of Manchester, 2009.

Lee and

Akin, Augmented reality-based computational fieldwork support for equipment operations and maintenance, Automation in Construction20(4) (2011), 338–352. doi:10.1016/j.autcon.2010.11.004.

Long,

Kooper,

Abowd and

Atkeson, Rapid prototyping of mobile context-aware applications: The cyberguide case study, in: Proc. of the 2nd Annual International Conference on Mobile Computing and Networking, ACM, Rye, New York, USA, 1996, pp. 97–107.

10.

Maes, How to do the right thing, Connection Science1(3) (1989), 291–323. doi:10.1080/09540098908915643.

11.

Marmasse and

Schmandt, Location-aware information delivery with ComMotion, in: Proc. of the 2nd International Symposium on Handheld and Ubiquitous Computing, Springer-Verlag, Bristol, UK, 2000, pp. 157–171. doi:10.1007/3-540-39959-3_12.

12.

Mileo,

Merico and

Bisiani, Support for context-aware monitoring in home healthcare, Journal of Ambient Intelligence and Smart Environments2(1) (2010), 49–66.

13.

Oh and

Woo, CAMAR: Context-aware mobile augmented reality in smart space, in: Proc. of International Workshop on Ubiquitous Virtual Reality, 2009, pp. 48–51.

14.

S.K.

Ong and

Zhu, A novel maintenance system for equipment serviceability improvement, CIRP Annals-Manufacturing Technology62(1) (2013), 39–42. doi:10.1016/j.cirp.2013.03.091.

15.

Qin,

Shi and

Suo, Ontology-based context-aware middleware for smart spaces, Tsinghua Science and Technology12(6) (2007), 707–713. doi:10.1016/S1007-0214(07)70179-7.

16.

B.N.

Schilit,

Adams and

Want, Context-aware computing applications, in: Proc. of the 1994 First Workshop on Mobile Computing Systems and Applications, IEEE Computer Society, 1994, pp. 85–90.

17.

Schmidt,

Asante and

Takaluoma, Advanced interaction in context, in: Proc. of the 1st International Symposium on Handheld and Ubiquitous Computing, Springer-Verlag, Karlsruhe, Germany, 1999, pp. 89–101. doi:10.1007/3-540-48157-5_10.

18.

Schwald and

DeLaval, An augmented reality system for training and assistance to maintenance in the industrial context, in: International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, UNION Agency-Science Press, Plzen, Czech Republic, 2003, pp. 425–432.

19.

Shehzad,

H.Q.

Ngo,

K.A.

Pham and

Lee, Formal modeling in context aware systems, in: Proc. of the First International Workshop on Modeling and Retrieval of Context, 2004.

20.

I.J.

Song and

S.B.

Cho, Bayesian and behavior networks for context-adaptive user interface in a ubiquitous home environment, Expert Systems with Applications40(5) (2013), 1827–1838. doi:10.1016/j.eswa.2012.09.019.

21.

G.P.

Sullivan,

Pugh,

A.P.

Melendez and

Hunt, O&M best practices – A guide to achieving operational efficiency (release 2.0), Pacific Northwest National Laboratory (Prepared for the Federal Energy Management Program U.S. Department of Energy), Richland, WA (US), 2004.

22.

D.W.F.

Van Krevelen and

Poelman, A survey of augmented reality technologies, applications and limitations, International Journal of Virtual Reality9(2) (2010), 1–20.

23.

Wang,

Feng,

Zeng and

Li, An augmented reality based system for remote collaborative maintenance instruction of complex products, in: 2014 IEEE International Conference on Automation Science and Engineering, IEEE, 2014, pp. 309–314. doi:10.1109/CoASE.2014.6899343.

24.

Wang,

Zhang,

Gu and

H.K.

Pung, Ontology based context modeling and reasoning using OWL, in: Proc. of the Second IEEE Annual Conference on Pervasive Computing and Communications Workshops, IEEE Computer Society, 2004, pp. 18–22. doi:10.1109/PERCOMW.2004.1276898.

25.

Zhu,

S.K.

Ong and

A.Y.C.

Nee, A context-aware augmented reality assisted maintenance system, International Journal of Computer Integrated Manufacturing28(2) (2015), 213–225. doi:10.1080/0951192X.2013.874589.

26.

Zhu,

S.K.

Ong and

A.Y.C.

Nee, A context-aware augmented reality system to assist the maintenance operators, International Journal on Interactive Design and Manufacturing8(4) (2014), 293–304. doi:10.1007/s12008-013-0199-7.

A novel context-aware augmented reality framework for maintenance systems

Abstract

Keywords

1. Introduction

2. Related work

3. System overview

3.1. Classification of information instances

3.1.1. Static information

3.1.2. Periodical information

3.1.3. Dynamic information

3.2. AR-based maintenance system

3.4.2. Context collector module

3.4.3. Reasoner module

4. Experiment

Table 1 Number of individuals Class Name No. of individuals Person 1 Device 1 Location 18 Equipment 20 Activity 1, 5, 10, 15, 25, 35, 50, 75, 100, 125, 150

References

Table 1
Number of individuals

Class Name No. of individuals

Person 1

Device 1

Location 18

Equipment 20

Activity 1, 5, 10, 15, 25, 35, 50, 75, 100, 125, 150