Active intrusion detection and prediction based on temporal big data analytics

7.1 The UNSW-NB15 dataset

IN 2015, Mustafa and Slay [19, 28, 29] created a dataset called UNSW-NB15 using an IXIA tool to extract a hybrid of attack activities synthesized between modern normal and contemporary modern network traffic, within from the UNSW Cyber Security Laboratory. A tcpdump tool (tcpdump tool, 2014) was used to capture 100 GB of raw network traffic, the Argus tools (Argus tool, 2014) and Bro-IDS (Bro-IDS tool, 2014) were configured in a process parallel to extract attributes [30].

The UNSW-NB15 dataset includes 49 attributes and 2,540,044 records stored in four CSV files [19, 30]-which are classified into six groups: the first group is stream attributes (Srcip, Sport, dstip, dsport, Proto) including identifier attributes between hosts, the second group is basic attributes (state, dur, sbytes, dbytes, Sttl, Dttl, sloss, dloss, service, sload, dload, spkts, dpkts) which implies the attributes representing the protocol connections, the third group is the content attributes (Swin, Dwin, Stcpb, Dtcpb, smeansz, dmeansz, trans depth, res bdy len) which encapsulate the TCP/IP attributes, it also contains certain attributes of http services, the fourth group is time attributes (sjit, djit, stime, ltime, sintpkt, dinpkt, tcprtt, synack, ackdat) which contains time attributes, the fifth group is additional generated attributes which can be split into two groups: general purpose attributes (Is sm ips ports, Ct state ttl, Ct w http mthd, Is ftp login, Ct ftp cmd) where each attribute has its own function, in order to protect the protocol service and the connection attributes (Ct srv src, Ct srv dest, Ct dst ltm, Ct src ltm, Ct src dport ltm, Ct dst sport ltm, Ct dst src ltm) which are constructed from the stream of 100 record connections based on the sequential order of the last time function. And the sixth group is the category of labeled attributes (Attack cat, Label) which are provided to label this dataset [19, 30].

The UNSW-NB15 dataset includes nine types of attacks which are: Fuzzers, Analysis, Backdoor, Dos, Exploit, Generic, Recognition, Shellcode and Worm [19, 28] (see Fig. 4).

•

Fuzzers: an attack in which the attacker tries to discover security holes in a program, an operating system or a network by feeding it with the massive capture of random data to make it crash [27, 30].

•

Analysis: an attack method that violates Internet applications via ports (for example, port scans), email (for example, spam) and web scripts (for example, HTML_les) [27].

•

Backdoor: a technique in which a system security mechanism is bypassed by stealth to access a computer or its data [29].

•

Back: an attack that disrupts computer resources via memory in order to prevent authorized requests from accessing a device [27].

•

Exploit: a sequence of instructions which takes advantage of a security problem within an operating system or software and exploits this knowledge by exploiting the vulnerability [27, 29].

•

Generic: a technique which establishes against each block encryption using a collision hash function without regard to the configuration of the block encryption [30].

OPEN IN VIEWER

Figure 4.

Attacks distribution in UNSW-NB15 database.

•

Recognition: an attack that gathers information on a computer network to escape its security controls [27, 30].

•

Shellcode: an attack in which the attacker enters a light piece of code from a shell to control the compromised machine [30].

•

Worm: a malicious program which requires user action to trigger and distribute it over a computer network by replicating itself according to the security failures of the target computer [27].

7.2 The CICIDS2017 dataset

The CICIDS2017 [45] dataset is a network intrusion detection dataset that was created by the Canadian Institute for Cybersecurity (CIC) at the University of New Brunswick. It was released in 2017 and is designed to be used for research purposes in the field of network security.

The dataset contains a total of 15 days’ worth of network traffic data, collected from a real-world network environment. The traffic was generated using a variety of network protocols, including HTTP, FTP, DNS, and SMTP, among others. The dataset includes both benign and malicious traffic, The CICIDS2017 dataset is a labeled network traffic dataset that contains various types of attacks as well as normal traffic. The dataset includes the following types of attacks:

•

Brute Force Attack: An attacker tries to guess a password by trying different combinations of usernames and passwords.

•

DoS Attack: An attacker floods a server or network with traffic to make it unavailable for legitimate users.

•

Web Attack: An attacker exploits vulnerabilities in web applications to gain unauthorized access or to perform other malicious activities.

•

Infiltration Attack: An attacker gains unauthorized access to a system or network by exploiting vulnerabilities or using social engineering techniques.

•

Botnet Attack: An attacker uses a network of compromised computers to perform malicious activities, such as sending spam or launching DDoS attacks.

•

DDoS Attack: An attacker uses a network of computers to flood a target system or network with traffic, making it unavailable for legitimate users.

•

PortScan Attack: An attacker scans a network for open ports and vulnerabilities that can be exploited.

OPEN IN VIEWER

Figure 5.

Attacks distribution in CICIDS2017 database.

•

Fuzzing Attack: An attacker sends random or invalid data to a system or network to find vulnerabilities.

•

MITM Attack: An attacker intercepts communication between two parties to gain access to sensitive information.

•

SSH Brute Force Attack: An attacker tries to guess SSH login credentials by trying different combinations of usernames and passwords.

The dataset contains a total of 2.8 million network flow records, with each record consisting of 80 features, including network flow characteristics such as duration, number of packets, and bytes transferred, as well as transport layer and application layer protocol fields.

UNSW-NB15 and CICIDS2017 datasets must meet critical characteristics, including data integrity, security, concurrency control, data recovery, scalability, performance, data consistency, flexibility, and extensibility. In our contribution, we harness the power of Azure and Spark technologies to reinforce these characteristics. Azure offers scalable infrastructure and robust data recovery options, enhancing scalability and data recovery in our database solution. Additionally, Spark enables efficient processing, bolstering performance and data analysis capabilities.

In the proposed contribution, both UNSW-NB15 and CICIDS2017 datasets are stored in DBFS, the distributed file system available on spark cluster in Databricks. The UNSW-NB15 dataset contains approximately 2.5 million instances and is approximately 2.4 GB in size, and the CICIDS2017 dataset contains approximately 2.8 million instances and is approximately 5.5 GB in size. The proposed system based on Apache Spark has to reduce the time of data analysis.

The proposed system has the following steps:

•

Collected files are uploaded directly to Azure Blob Storage.

•

A Spark job reads the data while it’s being placed in Azure Blob Storage.

•

Data is pre-processed and cleaned.

•

Data is classified using a Decision Tree pipeline.

•

Classifications are stored into Azure Data Lake Store.

This process involves collecting dataset files and loading them into Microsoft Azure Blob Storage. Blob Storage is built to meet HDFS standards and use global and local replication with Exabytes of capacity and massive scalability along with enterprise grade security. In order to obtain dataset files, we use the Microsoft Azure SDK for Python, which is a set of Python packages that make it easy to access components of Microsoft Azure.

The main steps of the proposed approach as shown in Fig. 6 can be summarized in these following steps:

1.

Filling missing values.

2.

Elimination of redundancies.

3.

Digitization of attributes of type String.

4.

Data classification based on Decision Tree model.

5.

Prediction based on Chronicle algorithm.

6.

Recommendations

7.3 Filling missing values

This step consists of filling in the missing values in the database records. Filling missing values involves replacing the missing values in a dataset with some appropriate values in order to improve the overall quality of the data and thus improve the accuracy of the analysis. To fill in the missing fields, the proposed solution is as follows: Each attribute of integer type will take 0 as a default value. On the other hand, attributes of type String will take “Unknown”. To perform this solution, we applied the na.fill() method. After applying this solution, the result obtained is a database with no missing values.

OPEN IN VIEWER

To address potential issues related to duplicate data in the dataset, a meticulous analysis was conducted. This analysis involved the utilization of the “dropDuplicates()” method, a technique that systematically identifies and removes redundant records [17]. By implementing this method, we achieved several essential outcomes. Firstly, it led to a substantial reduction in the dataset’s size, resulting in storage space savings and more efficient data processing. Secondly, the elimination of redundancies enhanced the accuracy and reliability of the data, which in turn reduced the time required for analysis and computations. Furthermore, this optimization step elevated the overall dataset quality, rendering it more suitable for subsequent data-driven tasks, including data analysis, modeling, and machine learning. Ultimately, these measures were instrumental in enhancing the overall detection rate performance of the system, ensuring its effectiveness in identifying and mitigating potential threats.

7.5 Digitization of attributes of type string

This step consists in digitizing the attributes of type String (converting them into digital or numerical format). This step Improves data analysis: By converting strings into digital format, it becomes possible to run queries and perform analysis on the data. It also allows efficient data storage since digital data takes up less storage space than text data. And allows faster processing, this means that digitized data can be processed in real-time, allowing for faster recommendations.

To perform this step, the StringIndexer () method is used.

After having stored and preprocessed, the data is hence ready to be used by the ML classification model. First of all, it is important to divide the data set in two parts: training data for model training and test data for performing the prediction.

The Machine Learning pipeline helps us bring together the stages of data transformation. After the creation of this class, a fit () method is called to start the transformation and learning from the training data, and a Transform () method to perform the prediction on the test data.

After classifying intrusions, comes the intrusion prediction part where the temporal analysis model is applied. Before applying this model, two columns are added to the database: the first one represents the difference between the two time attributes “Ltime” and “Stime” whereas the second one represents the time intervals noted t1, t2, …, tn.

After performing this preprocessing, the forecast model is applied to obtain an attack scenario in a time interval. Finally, to understand the quality of the predictions, we went through the evaluation stage where the calculation of the evaluation metrics is important.

7.6 Data classification based on decision tree model

There are many supervised machine learning algorithms, each with its own strengths and weaknesses. Here are some of the most common ones:

•

Linear regression: A simple algorithm used for predicting continuous values based on a linear relationship between input features and output variables.

•

Logistic regression: A classification algorithm that uses a logistic function to estimate the probability of belonging to a certain class.

•

K-nearest neighbors (KNN): A non-parametric algorithm that predicts the label of a sample based on the majority vote of its K-nearest neighbors in the training data.

•

Support vector machine (SVM): A binary classification algorithm that finds the hyperplane that maximizes the margin between the two classes in the feature space.

•

Naive Bayes: A probabilistic algorithm that uses Bayes’ theorem to estimate the conditional probability of belonging to a certain class, given the input features.

•

Gradient boosting: An ensemble algorithm that sequentially adds decision trees to the model, each one correcting the errors of the previous tree, to improve the model’s accuracy.

•

Decision tree: A tree-based algorithm that recursively partitions the data based on the values of input features, until the data is split into homogeneous subsets that are then assigned a label or class.

•

Random forest: An ensemble algorithm that combines multiple decision trees to improve the model’s accuracy and robustness.

In this contribution we opted for Decision trees as they have several advantages that make them popular for many applications. Here are some advantages of decision trees compared to the other machine learning algorithms:

•

Easy to understand and interpret: Decision trees are a highly interpretable model, making it easy for humans to understand how the model works and why it makes certain decisions.

•

Can handle both categorical and numerical data: Decision trees can handle both categorical and numerical data without the need for feature scaling, making it a versatile algorithm.

•

Can handle missing data: Decision trees can handle missing data by simply skipping over that feature when making a split, making it robust to missing data.

•

Can handle non-linear relationships: Decision trees can capture non-linear relationships between features and the target variable, making it a useful algorithm when dealing with complex datasets.

•

Fast to train and classify: Decision trees have a relatively fast training time and can quickly classify new data points once the tree has been built.

•

Can be used for feature selection: Decision trees can be used for feature selection by identifying the most important features for predicting the target variable.

Decision trees are a simple and effective supervised machine learning algorithm. In fact, a decision tree can be represented with nodes and edges. It is, also, composed of a root node which performs the first split nodes and leaves in which we can find the expected results [17].

Decision trees [36] are one type of supervised machine learning. They are defined by two entities: the nodes and the leaves where the decision nodes are where the data is divided and the leaves represent the final results.

The decision trees [37] are used to classify the examples according to the values of their attributes. Each attribute becomes a node in the constructed tree and their possible values determine the paths. The heuristics used to measure the purity of learning data are:

•

Entropy: Is the measure of the amount of uncertainty or chance in data. This metric provides information on the predictability of a certain event. The purpose of entropy is to measure the discriminating power of an attribute for the classification task. Here is the formula for entropy impurity:

B ⁢ ( k ) = - ∑ 1 d p ⁢ ( 𝐾𝑎 ) ∗ log ⁡ 2 ⁢ ( p ⁢ ( 𝐾𝑎 ) )(1)

With k an attribute for the decision tree.

•

Information gain: Is the actual change in entropy after deciding on a particular attribute. It measures the relative change in entropy with respect to the independent variables. The purpose of this metric is to rank the filter attribute at the given node in the tree. Here is the formula for the Information gain measure:

G ⁢ ( k ) = - ∑ i = 1 J p ⁢ ( i ) ∗ ( 1 - p ⁢ ( i ) )(2)

With P ⁢ ( i ) the probability of a classification i .

Ranking in the decision tree is based on high information gain entropy in descending order.

After storing and preprocessing the data, our data is ready to be used by the “Decision Tree” classification model. First, it is important to divide the data set into two: training data for model training and test data for making the prediction. Then, we have a machine learning pipeline that helps us bring the stages of data transformation together. After this class is created, a Fit () method is called to start the transformation and training of training data, and a Transform () method to perform prediction on the test data.

7.7 Prediction based on chronicle temporal model

The tools for temporal analysis are very limited in the literature. We mainly cite:

ARIMA: (Autoregressive Integrated Moving Average) [16] is a prediction model based on linear functions; Its goal is to find a simple model that can provide a sufficient description of the observed data. This model is used to predict future values of the time series at time t using past values.

ARIMA is a widely used statistical model for analyzing and forecasting time-series data. However, there are several drawbacks to the ARIMA model that should be considered:

•

Stationarity Assumption: ARIMA assumes that the time series is stationary, meaning that its statistical properties (mean, variance, etc.) do not change over time. In practice, many time series are non-stationary, meaning they have a trend, seasonality, or other patterns that change over time. ARIMA models can be extended to handle some types of non-stationary data, but it requires additional model components or transformations.

•

Lag Selection: The performance of ARIMA models depends on selecting the appropriate lags for the autoregressive (AR) and moving average (MA) components. Choosing the wrong lags can lead to poor model performance and inaccurate forecasts. The process of selecting lags can be time-consuming and requires domain expertise and statistical knowledge.

•

Data Requirements: ARIMA models require a minimum amount of data to estimate the model parameters accurately. The more complex the model, the more data it requires. If the time series is short or has missing data, ARIMA models may not be suitable.

•

Noisy Data: ARIMA models assume that the time series data is generated by a stochastic process with a specific mathematical structure. However, real-world time series data is often noisy, and it may not follow the assumptions of the ARIMA model. In such cases, the ARIMA model may not provide accurate forecasts.

•

Complexity: ARIMA models can become very complex when dealing with multiple AR and MA components, seasonal patterns, and other variations. This complexity can make it challenging to interpret the model results and communicate them effectively to stakeholders. Moreover, the model may be difficult to maintain and update as new data becomes available.

Recurrent Neural Networks (RNNs): A recurrent neural network [16] is a model that can model and predict complex nonlinear ones. Thanks to the cycles in its interconnection diagram, the model has internal memory. In theory, the network can memorize displayed information in an arbitrarily high order, but practically, in the long run, it will lose its efficiency in dynamic processes.

RNNs are a class of neural networks that are designed to handle sequential data such as time-series or natural language. While RNNs have been successful in various applications, they also have some drawbacks that should be considered:

•

Vanishing and Exploding Gradients: RNNs are prone to vanishing and exploding gradients, which can make it difficult to train the model. This happens when the gradient signal propagated through time becomes too small or too large, respectively. This problem can be addressed by using special types of RNNs, such as Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU), which can better handle long-term dependencies.

•

Memory Constraints: RNNs have a memory constraint, which means that they can only store a limited amount of information from the past. This can be a problem when dealing with long sequences, as the network may forget important information from earlier in the sequence.

•

Computational Complexity: RNNs can be computationally expensive to train and evaluate, especially when dealing with long sequences or large datasets. This can limit their applicability in some domains.

•

Input and Output Constraints: RNNs require that input and output sequences have a fixed length. This can be a problem when dealing with variable-length sequences, such as natural language sentences. Techniques such as padding or truncation can be used to overcome this problem, but they may introduce additional noise in the data.

•

Limited Parallelism: RNNs are inherently sequential models, which means that they cannot be easily parallelized. This can limit their speed and scalability, especially when dealing with large datasets.

•

Interpretability: RNNs can be difficult to interpret, as the network weights and activations do not have a direct interpretation in terms of the input data. This can make it challenging to understand how the model makes its predictions.

Chronicles: A chronicle is a set of events linked together by time constraints representing a scenario of attacks. This model takes the date event stream as input and identifies chronicle instances as events develop. One of its main advantages is that it is possible to make a complete forecast of the possible dates for each expected event [16].

The Chronicle temporal analysis model is used in this contribution as it is a powerful tool for identifying patterns and anomalies in large datasets. When applied to cybersecurity, it can be used to predict potential intrusions by analyzing historical data to identify patterns of behavior that could indicate an attack. It has several advantages:

•

Multi-granular Analysis: The Chronicle model allows for the analysis of time-series data at multiple levels of granularity, such as yearly, monthly, weekly, daily, or even hourly. This can help identify patterns and trends that may not be apparent at coarser time scales.

•

Temporal Context: The Chronicle model emphasizes the importance of temporal context, which means that the analysis takes into account the sequence of events and their timing. This can help identify the causal relationships between events and understand the impact of time on the data.

•

Event-centric Analysis: The Chronicle model focuses on events as the primary unit of analysis, rather than individual data points. This can help identify meaningful patterns and trends that are related to specific events or sequences of events.

•

Interactive Visualizations: The Chronicle model includes interactive visualizations that allow users to explore and analyze the data in real-time. This can help users gain insights and identify patterns quickly and efficiently.

•

Multiple Perspectives: The Chronicle model allows for the analysis of time-series data from multiple perspectives, such as statistical, spatial, or semantic. This can help identify patterns and trends that may not be apparent when analyzing the data from a single perspective.

•

Scalability: The Chronicle model is scalable, which means that it can be applied to large datasets without sacrificing performance. This can be particularly useful when dealing with high-frequency or streaming data.

The Chronicle temporal analysis model, Recurrent Neural Networks (RNNs), and Autoregressive Integrated Moving Average (ARIMA) are all useful tools for analyzing and forecasting time-series data, but they have different strengths and weaknesses.

The Chronicle model is particularly well-suited for analyzing complex, multi-dimensional time-series data, such as data from social media, sensor networks, or financial markets. It emphasizes the importance of temporal context, event-centric analysis, and interactive visualizations, which can help identify patterns and trends that may not be apparent using other methods. The Chronicle model can also handle missing or noisy data and is scalable to large datasets.

On the other hand, RNNs and ARIMA are more focused on forecasting future values based on past values of a single time series. RNNs are particularly well-suited for handling sequential data with long-term dependencies, such as natural language or financial data, and can handle variable-length inputs and outputs. ARIMA is a popular statistical method for modeling stationary time series, such as economic or financial data, and can be easily interpreted and explained.

Therefore, the choice between these models depends on the specific requirements of the analytical task. If the goal is to analyze complex, multi-dimensional time-series data and identify patterns and trends over time which is the case here, the Chronicle model may be the best choice. If the goal is to forecast future values of a single time series, RNNs or ARIMA may be more appropriate depending on the nature of the data [33].

In this proposed contribution, to predict intrusions, a Chronicle model, which is represented in the form of a constraint graph where the events are represented by nodes and the temporal constraints are the labels connecting the nodes, is used. A chronicle system is made up of three parts:

•

A set of predicates

•

A set of temporal constraints concerning these predicates

•

A set of actions to be applied when the chronicle is recognized (optional).

OPEN IN VIEWER

Figure 7.

Chronicle example.

After classifying the intrusions, we move on to predicting them by applying the “Chronicle” algorithm after adding two columns to the database: the first column represents the difference between two time attributes “Ltime” and “Stime” and the second column represents the time intervals noted “ Δ t1, Δ t2, …, Δ tn”. After performing this preprocessing, the “Chronicle” algorithmis applied to obtain the attack scenarios in the time intervals.

7.8 Recommendations

Finally, to provide recommendations, we went through the forward chaining algorithm which calculates the recommendations that better fit the detected attacks.

In scenarios where a different approach is employed, such as backward chaining, the inference engine would start with a goal or conclusion and work backward to determine which rules or facts support that goal. The choice between forward and backward chaining often hinges on the nature of the problem, the availability of data, and the desired outcome.

The forward chaining is a mode of reasoning the inference engine of an expert system. This is a method of rules deduction starting from the premise to derive new conclusions. These findings enrich the working memory and can become the other rules premises. Forward chaining is a method of reasoning data driven, unlike the backward chaining who shares the findings to try to go back to the axioms. In the forward chaining mode, the inference engine share from facts to reach the goal, that is to say, it selects only the rules whose conditions of the left are checked (of the base facts), and then applies the first rule, which adds the conclusion to the base.

One of the advantages of the forward chaining algorithm on the backward chaining is that the receipt of new data may cause new inferences, making the inference engine best suited to dynamic situations in which the conditions are likely to change. The forward chaining is used in artificial intelligence, in a rule-based expert system, a rules engine, or in a production system.

To complete our presentation of forward chaining, the algorithm is presented in Fig. 8.

OPEN IN VIEWER

The forward chaining algorithm is a bottom-up approach that starts with a set of initial facts and uses the rules to infer additional facts until a goal is reached. Here are the general steps involved in the forward chaining algorithm:

•

Initialize the working memory with a set of initial facts.

In summary, the forward chaining algorithm works by iteratively applying rules to infer new facts and adding them to the working memory until a goal is reached or there are no more rules to fire.

The decision to employ an Expert System, specifically utilizing the forward chaining algorithm, rather than alternative approaches like Intelligent Agents, in our intrusion detection and recommendation system stems from its compatibility with rule-based decision-making in the realm of cybersecurity. An Expert System with forward chaining is well-suited for making recommendations and decisions based on predefined rules and expert knowledge, crucial for handling potential cyber threats effectively. Furthermore, the forward chaining algorithm’s adaptability to changing conditions and its ability to build conclusions step by step as new data is received make it particularly suitable for dynamic and real-time intrusion scenarios. Its established utility in artificial intelligence and expert systems adds to its credibility, providing transparency and traceability in decision-making, essential for auditing and accountability in security-related applications.