An intelligent self-adaption complex event processing framework with dynamic context detection and automatic event pattern modification abilities

2.1 Large-scale dynamic monitoring scenario

Most studies on CEP’s application in the retail industry focus on anti-theft problem [9]. Further application was described in [9] which studied the monitoring of active inventory level. It pointed out that special CEP rules could be developed to cope with unexpected circumstances due to changes of external environment. On the basis of this research, this paper further explores the application of CEP in large supermarkets.

In a large supermarket, there are usually at least several thousand kinds of goods, which have different turnover periods. For example, carbonated drinks can be kept for one year; and fish and fruit can only be kept for one week. Meanwhile, sales of goods fluctuate due to season, promotions and other special factors. For example, cool drinks are sold well in summer; and sales of electric blankets and other heating products rise in winter. So, the inventory level of the thousands of goods needs to be adjusted constantly. It is obviously a problem of large-scale dynamic monitoring to maintain the optimum inventory level of each commodity. Automated methods must be used to solve such a problem. However, current supermarket management systems usually combine systems and manual methods to determine inventory level and ordering frequency. This kind of practice is clearly not real-time and responds in a lagged way.

Introduction of CEP can automate business processing and thereby alleviate some of the above problems. For example, goods’ inventory level can be monitored by CEP in a real-time manner, and the manager will be informed of immediately if any goods’ inventory level is less than 10% . As inventory level changes quickly, traditional CEP actually can’t fully meet the business needs. For example, for period with weekly average temperature exceeding 30°C, the sales of summer sleeping mat will increase rapidly and CEP engine can adjust the parameter threshold of event pattern (i.e., inventory monitoring) to 20% automatically to avoid shortages of goods; and for period with weekly average temperature lower than 10°C, the parameter threshold should be adjusted to 5% to avoid overstock. It is a tedious task to judge appropriate event pattern under related conditions and set the correct parameters by domain experts. Extending traditional CEP with a self-adapting ability is a first-choice solution.

2.2 Adaptive CEP’s tasks

Key to the function of CEP engine is event pattern (also called event rules). Event pattern consists of pattern parameters and pattern structure. Pattern parameters include variables constraining value of single event, variables comparing events, window of time and so on. As event pattern is linear, there are two important aspects in pattern structure, event operator type and pattern length (i.e., the number of operators used). For example, the pattern length of R1 (A seq B seq C) is 2. Based on it, this paper gives the definition of adaptive CEP first to determine the specific research of this direction.

Definition 1. CEP’s adaptability refers to its ability to automatically choose the structure or constitution parameters to be executed according to context.

There are factors from both external and internal systems to promote event pattern’s adaptation. External context (i.e., environmental conditions), which is intuitive, usually works as the basis of event pattern. For example, a large supermarket’s main activities, purchasing, selling and storing goods, are determined by external factors such as seasons, holidays and weather, though they are not involved directly. Internal context is complex and often cannot be obtained directly. Experts in the field are needed to do analysis and modeling to get it. As shown in Fig. 1, if sales of a commodity keep increasing at a rate of 10% during a certain period (e.g., event B occurs three times sequentially), CEP engine will automatically increase related event pattern’s parameter threshold accordingly, thus realizing real-time monitoring of current commodity inventory level in the supermarket management system. To determine whether the context appears, original events (event E, only including data such as commodity name, sales, marketing time, etc.) from POS (Point of Sale) should be aggregated into event A, sales of commodity in a reasonable set window of time. Then, event B, fluctuation in the sales volume of the commodity, can be generated by detecting pattern (increase or decrease) of event A. Finally, instructions on adjusting related event pattern parameter can be generated by matching the pattern of event B. It can be observed from this process that the judgment of internal context also requires matching of event pattern. And event pattern of adaptive CEP can be divided into two types accordingly, one type for decision (called conditional event pattern) and one type for executing actual activities (called business event pattern).

In summary, the adaptive CEP system should not only be able to deal with environmental information from external event streams, but also be able to observe its internal changes, and thus timely adjust parameters and structure of business event patterns to adapt to their current context.

3.1 Context detection module

This module is used to describe and detect dynamic contexts (including internal and external disturbances). To describe the context, modeling of input data is needed first. It can be observed from Fig. 2 that system accepts two kinds of data sources, business activities and environmental information event sources (e.g., temperature events released periodically by the weather forecast web service). To standardize the format of event sources for unified processing, based on an object-oriented model, we assume event data from event sources is a light-weight data tuple with a schema defined as a list of attribute name and type pairs, i.e.,

Event: = Event_Type @ (Attribute: Data_Type*)

Where, Event_Type refers to a kind of event object with the same characteristics; attribute is an integral element of event, and an event must contain at least one attribute; Data_Type represents the data type of event attribute.

Although it’s not included in the previous literature, we identify that data type is necessary since computer processing, as well as structural modeling, shall also be taken into account. When an event type is given a value, according to the data type of attributes, it is called an event instance. And event stream can be produced by sending event instances according to time sequence.

Context detection among event streams includes event pattern evaluation and event semantic inference. For example, the summer in western China includes months from May to September; when the system receives events with temperature over 30°c, it can deduct that it is summer by using semantic information contained in the event (such as time and location) as well as background knowledge. Therefore, the context should be described combining event pattern description and event’s semantic description. Based on the pipeline concept proposed in [10], this paper extends TESLA (Trio-based Event Specification Language) proposed in [3], which can describe event pattern in detail and flexibly, and combines it with SPARQL (Simple Protocol and RDF Query Language) to form a SCEP rule language, which is able to declare semantics to meet the need to describe patterns in dynamic context. The model is defined as follows:

Intuitively, SCEP rule language consists of CEP sub-pattern and semantic sub-pattern. Square brackets are used to identify them. CEP sub-pattern consists of a number of clauses with key words. Select clause defines the structure exporting complex events. From clause defines the event pattern composed of simple events. Where clause defines attribute constraints of atomic events. Define clause defines actual attribute value of complex events. Semantic sub-pattern includes rules in addition to SPARQL queries to realize profound semantic reasoning of events. The following SCEP rule R1 is the specific declaration of the example in this Section:

Execution of R1 is determined by the internal structure of the context detection module shown in Fig. 2, and the specific process of it is shown in Fig. 3. First, original event streams will flow through SCEP processing engine (left in Fig. 3) and form complex events after attribute filtering, time sequence detection, aggregation calculation and other operations. Then, the complex event consisting of all the component events will be sent to semantic event engine (right in Fig. 3), which will perform semantic annotation (upper right side in Fig. 3) and semantic inference (Fig. 3 lower right) of events. The lower right part of Fig. 3 is a fragment of ontology knowledge base, which contains some ontology concepts such as the event, city, month, and season. Once those concepts and the annotation event instance are combined, inference engine can immediately derive implicit knowledge using rules (e.g., when the ontology in Fig. 3 and the rules in R1 are combined, it can be inferred that event:ID1234 occurred in summer). Finally, the generated business event with high-level semantics will be sent to the nextmodule.

3.2 Event pattern object identification module

This module is used to identify affected business object based on the semantics of input events, so as to determine the parameter values or structure of related event pattern to be adjusted. Its internal implementation adopts two libraries (i.e., rule library and decision model library). So, it can be regarded as a standard two-library intelligent decision support system.

Rule library is used to store business background knowledge predefined by domain experts according to their experience. It can support the business rule processing engine. The engine proposed in this paper is based on Drools [11]. Therefore, the business rules are described using Drools’ rule language. For example, the following rules describe the business logic in the case that Coca-Cola’s sales are expected to rise 60% in summer (e1) and also during the International Workers’ Day (e2).

Decision model library is used to store prediction model, analysis model and other models for decision making, and can be invocated by model execution engine. Its exported results include parameters and structure of related event pattern (for details, see Section 3.3). Each decision model in the library will be implemented as a code class including calculation process. It can be seen that decision model library is actually a set of program class. Invocation of model is also described using Drools’ rule language. An example is given below.

It can be seen that, in addition to sufficient flexibility for loose coupling between invocation logic and code logic, description of invocation relation in such a way can maintain an event-driven approach, facilitating integration with the previous processing engine.

3.3 Event pattern adjustment module

This module is used for real-time monitoring of changes in parameters and structure of event pattern and performing updating accordingly. Its core is event pattern’s XML configuration file, which mainly includes two types of nodes (as shown below, only parts of them are shown for limited space). The first type is adaptive parameter nodes (labeled as <adaptive-param>), which can inject dynamic parameter values into the event pattern. Specific input values are first initialized by the user, and then adjusted automatically by decision model execution engine (see Section 3.2). The second type is the event structure nodes (labeled as <pattern-structure>). A structure node is actually an event pattern. A complete event pattern can be divided into a number of sub-patterns. For example, P1 = (A seq B seq C) can be divided into P2 = (A seq B) and P3 = (P2 seq C). Therefore, a structure node typically consists of at least one sub-pattern node (labeled as <sub-pattern-node>). The following example is given to check whether the inventory level of Coca-Cola reaches its minimum level. Its event pattern contains only one sub-pattern node. Sub-pattern is described by Petri-net elements. (Implementation of CEP by Petri-net is not covered by this study. For more information, please refer to [6, 7].) And the parameter values, as the component elements of sub-pattern, can directly refer to the defined parameters nodes (labeled as <ref adaptive-param>).

This module works in the following way: First, the event pattern declared by SCEP rule language is converted into configuration file via an XML parser. The conversion focuses on searching the parameter nodes with related id attribute value in configuration file according to variable name in the declaration, and then injecting them into the structure nodes through its reference (labeled as <ref>). Then, notifications of events are received through the adaptive event monitor. Once there is a need to modify the value of a parameter node, or a sub-pattern node is to be added or deleted, the XML configuration file and related event pattern in CEP business execution engine will be updated immediately.

3.4 CEP business execution engine

In the actual CEP, event pattern parameters are usually involved in two types of comparison operations, single-attribute value comparison (e.g., A.inventory-value < “Coca-cola-minimun-par”) and dual-attribute value comparison (e.g., A.Timestamp >B.Timestamp). The former operation is completely static, and its output will not change with computation position. Meanwhile, as it is an element changing frequently during CEP’s adaptation, it can be separated from the CEP detecting network. A single attribute value’s comparison expression can be separated to form a filter node. The filter node links for the same class of events can form a filter chain. And the entire filter chain can be indexed to form an event pattern parameter pre-filter-net (Fig. 4). Filter-net is organized hierarchically. The most important node will be placed in the highest layer. For example, a filter chain consisting of three-layer filter nodes exists before place “a” in the example of Section 3.3. The first layer is the root filter node of event type (e.g., A.event-type=“Current-inventory”, the solid black point in the Fig. 4). Thus, when events are sent from the event source to CEP business execution engine, the atomic events will be filtered by all the root filter nodes in index, and then go through the filter chain, and are ultimately routed to the corresponding place in Petri detection net. If there is no related event pattern parameter, the root filter node will directly connect with the place.

Parameter pre-filter-net has at least two advantages. First, when the adaptive parameter node in event pattern configuration file (see Section 3.3) is updated, only member variable values of related filter nodes need to be reset. The process is easy and quick to execute. Second, parameters and structure of an event pattern can be separated into parts which are loosely coupled during adaptive process, facilitating event pattern’s adaptive adjustment.

3.5 User management tool

Finally, a specialized user management tool is provided in addition to AC-CEPs, which integrates document management and editing features. It is mainly used to organize the decentralized rules and configuration files in the system, making them easy for users to view. It also provides an editing interface for users to define declarations, rules and ontology knowledge.

4.1 Monitoring dynamic context

As shown in Fig. 2, the operation of context detection module involves SCEP engine, semantic annotation and semantic inference. First, colored Petri-net was used to run SCEP engine. Place received events, transition performed operator computation and arc constructed the network. Then, JenaBean [12] was used to realize semantic annotation, converting event object (Java object) to semantic object (RDF instance). Finally, Jena [13] Semantic Web framework was used to build event semantic inference engine. To meet the need of semantic explanation and reasoning, an ontology knowledge repository was provided in this module, which was described using OWL language, and organized with modular hierarchical approach proposed in [14].

Behaviors of context detection module were declared using SCEP rule language discussed in Section 3.1. SCEP engine performed CEP sub-pattern, semantic inference engine executed semantic sub-pattern, and semantic annotation feature linked these two components. This set of experiments tested the three parts separately using event pattern R1 mentioned in Section 3.1 with an objective to identify the key factors influencing the module’s efficiency.

The main workload of SCEP engine came from the calculation on place, transition and arc, which were mapped to the WHERE, FROM, and DEFINE clauses in CEP sub-pattern, respectively. Therefore, in order to see the performance of each part, only SELECT and DEFINE clauses were executed firstly, that is, all atomic events directly flowed through the Petri-net to generate complex events; then WHERE clause was added, that is, event filtering was added in the place; and finally, FROM clause was added, that is, events were aggregated in the place. As shown in Fig. 5(a), the transportation of events in Petri-net and encapsulation of complex events cost most of the processing time (up to 75%), while event attributes’ filtering and calculation of events’ relationship took a shorter time.

As shown in Fig. 5(b), processing duration of event semantic annotation completely depended on the length of event attributes. The time required for event conversion increased linearly with the number of attributes.

Experiments on semantic inference engine adopted the same multi-part testing method. First, the ontology (in Fig. 3) and the rules R1 (in Section 3.1) were built, and the engine was evaluated based on them; then, one hundred ontology classes were added in the ontology knowledge repository; subsequently, one hundred rules were added in inference rule file to test; finally, one hundred ontology classes and 100 rules were added at the same time for test. As shown in Fig. 5(c), increase in the number of ontology took up little processing time, while increase of inference rules increased processing time significantly. So, inference rule file can be classified according to event objects to optimize the efficiency of this module (i.e., calling related rule files according to event type).

4.2 Event pattern object identification

Since both the two processing engines in event pattern object identification module are based on Drools, the efficiency of this module is actually determined by the Drools’ processing capability. Drools’ implementation is based on optimized RETE algorithm, with a higher efficiency. In addition, it is object-oriented, expressed naturally and easy to adjust. To test the application of Drools, in this set of experiments, the module received two types of events input by event semantic inference engine (i.e., seasonal events and holiday event, and the event attribute length is 5), and the two engines ran the two sample rules mentioned in Section 3.2 separately. Both rules adopted consumption strategy (i.e., using retract function to consume matched facts from the working memory) to improve the efficiency of rules’ matching.

Figure 6(a) shows independent operation of business rule processing engine. It can be concluded that engine’s processing time increased linearly with the number of registered rules. In addition, as the two engines in the module operate in the same way with similar performance, the processing capability of decision model execution engine can be seen in Fig. 6(a). Figure 6(b) shows the two groups of data, which represented the processing time of two engines separately and the whole module when there was 1 or 100 registered rules and invoking decision models, respectively. The results showed that, the running speed of this module directly depended on the number of business rules and decision models. The processing time increased with the number of rules and models. Second, the efficiency of the overall module was better than that when operating two engines independently, as there was a working memory shared by the modules and an event-driven mechanism which can effectively avoid time cost for events transmission between two engines and insert operation using workingmemory.

4.3 Event pattern’s self-adaptation

Event pattern adjustment module monitors events from previous module as inputs. There are two types of events, parameter modification and structural adjustment. Two sets of event streams, each of which contains only one type of event, were simulated and sent at the speed of 1,000–10,000 events/sec for this group of experiments.

Figure 7(a) shows the results with parameter modification event stream. These events were simple in terms of structure, only containing parameter names and values. They were used to adjust the event parameters in event pattern. To check event parameter filter-net’s positive effect on efficiency (for details, see Section 3.4), experiment was carried out to compare the results of standard and pre-filter-net petri-net. In the experiment, each place included three event parameters (i.e., event type checking, equivalent comparison and non-equivalent comparison). These events parameters placed in filter-net formed a three-layer routing filter chain. In addition, as the number of event patterns in Petri-net has an impact on adaption, two types of petri-net were tested when there was 1 or 50 event patterns, respectively. It can be concluded from Fig. 7(a) that: first, the filter-net can substantially reduce the overall processing time (10 times faster at the rate of 10,000 events/sec); second, processing time to adjust event parameters in standard detecting network increases rapidly with the number of event patterns (up to 6.8 times), while the processing time increases slowly when using filter-net (up to 1.2 times). The reason is that all place nodes in standard detecting network need to be gone through and all internal arrays (data structure) should be searched to find the related event parameter. When a filter-net is used, as all event parameter nodes are stored in the Hash table, there is no need to search all the place nodes, thus saving time and eliminating the impact of increasing event patterns.

Figure 7(c) shows the results with structural adjustment event stream. These events have complex structure, and are generally required to be encapsulated hierarchically. It will cost some time to transform them into basic Petri-net structure. Also, as XML configuration file needs to be retrieved to find the previous connected place during a new sub-pattern insertion, operating performance of XML document is also evaluated. The abscissa of Fig. 7(b) indicates the total number of subpattern node (i.e., the total number of transition) in XML configuration file, and the ordinate represents the performance of the operation on this document. The results showed that it cost a longer time to search in document, which was parsed using DOM4j, and the processing time increased with the number of nodes in document. The time required to insert data in document was negligible and not influenced by the number of nodes. Because adjustment of event pattern’s structure is in three steps, generating Petri-net basic structure, searching XML configuration file and adding to petri detection network, the final experiment of this group compared the performance of each section under different workload. The abscissa of Fig. 7(c) indicates the input rate of event stream, and the ordinate represents the occupation proportion of three parts in total processing time of this module. By analyzing the results, we found that the first two steps accounted for about 85% of the processing time and less time was needed to do adjustment. The proportion of each step’s time was almost independent of input rate.