Host-based misuse intrusion detection using PCA feature extraction and kNN classification algorithms

Abstract

This paper presents the design and performance evaluation of a host-based misuse intrusion detection system for the Linux operating system. The proposed system employs a feature extraction technique based on principal component analysis, which is called Eigentraces, of operating system call trace data, and k-nearest neighbor algorithm for classification. The design is evaluated on the ADFA-LD dataset which entails one normal and six attack classes. Feature vectors are formed from fixed-size system call trace raw data through windowing and the principal component analysis, and serve as templates for the training phase. Classification of system call trace data that is in the form of feature vectors which are formulated through the Eigentraces procedure is accomplished using the k-nearest-neighbor algorithm. Two variants of the misuse intrusion detection system designs were evaluated through a simulation study on the ADFA-LD dataset: one design considered only two classes, namely normal and attack classes while the second design considered seven classes, namely one normal and six attack classes. In both cases the proposed design demonstrated very high performance. In overall, the misuse intrusion detection system was able to detect the attacks and predict the type of the attacks.

Keywords

Host based intrusion detection misuse detection principal component analysis k nearest neighbor classifier operating system call trace Linux operating system

1. Introduction

Internet and computer networks are inseparable aspects of our daily routines. Managing banking transactions online, exchanging sensitive private and confidential data over networks, and many other online activities are all ubiquitous and need preservation of privacy, confidentiality and protection against all kinds of exploitation. There does not appear to be a shortage of criminals attempting to access online data, accounts and sensitive information repositories with malicious intentions. Consequently, detection of and protection against such intrusions are critical for online infrastructure to play its valuable role in our lives.

Intrusion Detection Systems (IDS) are built using a combination of hardware and software, and as their name implies, employed to detect intrusions into computing systems. IDS designs are classified into three major categories, namely Signature-based Detection (SD), Anomaly-based Detection (AD), and Stateful Protocol Analysis (SPA) [1]. An SD type IDS detects the attacks and intruders by monitoring the activities throughout a network or a system, and identifying the pre-determined signatures of abnormal activities. As such, it is a useful method to detect the known attacks on a specific system. An AD type IDS generates a statistical model of normal behavior by monitoring the process during an attack-free period. It considers any activity out of normal range as an anomaly. It is an efficient method to detect zero-day attacks and it is not heavily dependent on the type of computing system. It may suffer from false alarms if the normal behavior is not covered or captured comprehensively as it will declare some normal behavior as anomalous. An SPA type IDS, which is also known as the deep packet inspection, is a resource intensive technique, which relies on vendor-developed universal profiles. Vendor-developed universal profiles determine the function of a well-defined protocol. In other words, it defines how a specific protocol should be and should not be used. For example, it can detect the length of command or the number of repeating a command and also the variation of minimum and maximum values for attributes and any other misbehavior. Unusual or unexpected activity outside the established profile is considered as an intrusion threat [2].

The signature-based intrusion detection technique is useful when there exist a proper, new and up-to-date dataset because type of threats are changing over time and detection process should be able to detect the most recent instance of malicious activities. Furthermore, for effective detection, attack signatures need to be extracted accurately from raw data. There are several techniques to extract the signatures from raw data. One prominent technique is the Principal Component Analysis (PCA) [3]. The Eigenfaces methodology which is based on PCA is a signature-based algorithm for face image analysis. It demonstrated applicability and superior performance for pattern recognition through dimension reduction and feature extraction [4]. Eigenfaces extracts the major features of a face image and encodes them as templates for training a classification algorithm [5]. Numerous studies about Eigenfaces report high performance for face recognition applications [5, 6, 7, 8].

In the Eigenfaces based approach, face images are represented as sequences of pixels each of which is a numeral. Pixel values are correlated in relation to their distances as measured through an appropriate distance metric. If two pixels are neighbors then their gray-scale or color-coded values are likely to be close to each other in most cases unless they are on the opposite sides of a very sharp boundary. Traces of system calls by applications concurrently executing in an operating system environment are somewhat similar to pixelated representation of images. Such a system call trace for multiple and concurrently executing applications will have subsequences of calls that will repeat throughout the period of capture for a given session. As system calls are represented as long sequences of non-negative integers with repeating subsequences, they exhibit similarities with raw pixelated representation of face images in some respect.

Studies on the design and development of misuse intrusion detection systems on network-level data are abound in the literature as cited in the next section while similar studies for detecting misuse intrusions at the operating system level are somewhat scarce. There is a need for studies to address this perceived gap in the design and development of misuse intrusion detection systems based on system call trace data at the operating system level. The study reported in this paper will present design and performance evaluation of a misuse intrusion detection system using a principal component analysis based technique, which is named as Eigentraces, for feature extraction, and the kNN algorithm for classifier model generation at the operating system level using system call traces of concurrently executing applications.

2. Related work on misuse detection

We next present a survey of recent works for misuse intrusion detection system (IDS) design and performance evaluation on several benchmark datasets in the associated domains.

Bhavesh Borisaniya [21] presented a misuse detection study on ADFA-LD and ADFA-WD datasets. They used the modified vector space along with the N-gram feature extraction technique and generated classifier models for both binary and multiple classes. They defined feature vectors using a multitude of N-grams, namely 1-gram, 2-gram, 3-gram and 5-gram. Among the several classifiers which they applied to the binary and multi class data sets, they reported an accuracy of 92% with 20% false positive rate (FPR) by using IBk and J48 classifiers (in Weka) as their best results for the multiclass model, and an accuracy of 96% with 19% FPR for the binary class problem.

In a study by Depren et al., authors reported on the development of a hybrid IDS architecture on the network-based KDD Cup 99 dataset [22]. The IDS design employs both misuse and anomaly detection. They used the C4.5 (J48 in Weka) decision tree algorithm to classify different attacks in datasets for misuse detection. For anomaly detection, the IDS design used a self-organizing map to model the normal mode of operation. They reported 99% accuracy and 0.2% FPR for the misuse detection system.

Assem et al. proposed a so-called SC2.2 misuse detection system for binary class problems using the UNM datasets [23]. Their design defines conditional probabilities using the Markov chain model of long sequences of system calls in four different UNM datasets. They reported the performance of the SC2.2 in terms of accuracy, true positive rate (TPR), and false positive rate (FPR) for naïve Bayes multinomial (NBm), C4.5 decision tree, Repeated Incremental Pruning to Produce Error Reduction (RIPPER), support vector machine (SVM), and logistic regression (LR) classifiers. Classifier accuracies are reported to be ranging from 97% to 99% with 0.3% to 3% FPR for all UNM datasets.

In another study that employed the UNM datasets, Kang et al. proposed a classifier model for binary class misuse detection [24]. They formulated fixed-length system call sequences using the STIDE windowing technique. They applied five different classifiers such as naïve Bayes, C4.5, RIPPER, SVM and LR for misuse detection through 10-fold cross-validation. They reported 100% accuracy with 0% FPR using the support vector machine as the best result.

Laskov et al. compared the performances of supervised and unsupervised learning algorithms in terms of detecting network-based both known and unknown attacks within the framework of binary classification [25]. Their data source consisted of the combination of instances from KDD Cup 1999, DARPA 1998 and DARPA 1999 data sets. For pre-processing, they split the attack data into disjoint subsets containing only one attack type and the normal data into disjoint subsets containing only one service type. They next merged these subsets to generate three equal length datasets for training, validation and testing. They performed metric embedding to transform the data into a metric space since KDD Cup dataset contains categorical and numerical features of various sources and scales. They experimented with many different supervised and unsupervised algorithms. The best result in performance evaluation for supervised misuse detection belongs to C4.5 with 95% TPR and 1% FPR. In terms of unsupervised learning, Single Linkage Clustering algorithm performed better than the others by scoring 70% TPR with 6% FPR. They also reported the performance for anomaly detection using two different classifiers: the SVM as a supervised method scored 80% TPR and 8% FPR while the Gamma algorithm produced 75% TPR and 7% FPR.

In a study reported by Kuchimanchi et al., authors proposed a dimension reduction approach using neural network-based feature extraction and CART classifier for real-time binary class network-based misuse detection system on the KDD 1999 dataset [26]. They performed neural network principal component analysis and nonlinear component analysis for feature detection and dimension reduction. The reported performance using the CART decision tree was 99% accuracy with 0.2% FPR.

As the presentation in this section indicates many researchers developed high performance misuse detection algorithms on a variety of datasets including several benchmarks. There are fundamental differences among these datasets. The KDD dataset and its derivatives possess network traffic data and not intended for host-based intrusion detection systems. The UNM trace data are sequences of system calls captured during execution of a single process. On the other hand, ADFA-LD dataset represents operating system call traces based on observations of a modern Linux local server offering file sharing, database services, remote access, and web server functionality among others.

Most designs are for the binary class case while a few designs for the multi-class case reportedly did not demonstrate high level performance. Among those listed, only one study reports a misuse intrusion detection system design on the ADFA-LD dataset which is also used in our study. Their misuse intrusion detection system’s performance is amenable to further improvement as they report relatively high false alarm rates of 20% for classification of both binary and multiple classes.

3. Data set and methodology

In this section we present the ADFA-LD dataset along with the data preprocessing, application of the PCA based Eigentraces methodology and machine learning classifiers employed in the IDS design.

3.1 ADFA-LD

Monitoring a process in a computer system using system-call trace sequences is a promising approach to detect malicious activities. ADFA-LD is a recent dataset which is collection of system call sequences and intended to help with the development of host-based intrusion detection systems [9]. Ubuntu Linux operating system, version 11.04, was the host for generating the ADFA-LD dataset. The operating system was reported to be fully patched and provided FTP, SSH and MYSQL version 14.14 services with their default ports. Apache version 2.2.17 running PHP version 5.3.5 was enabled to allow for web-based attacks. Also, TikiWiki version 8.1 was installed as a web-based collaborative tool. ADFA-LD dataset consists of seven different types of system call traces including normal and six web-based attacks, namely Adduser, Hydra-FTP, Hydra-SSH, Java-Meterpreter, Meterpreter and Web-Shell. Table 1 shows the attack types, effects and vectors.

ADFA-LD covers a pool of Linux system call traces under different conditions, which are compiled by the UNIX server security audit. It is categorized into three different groups, namely TRAINING_DATA_MASTER (TDM), VALIDATION_DATA_MASTER (VDM) and ATTACK_DATA_ MASTER (ADM). TDM and VDM are captured during normal processes ranging from web browsing to LATEX document preparation [10]. Table 2 presents these three ADFA-LD datasets with the type and count of traces for each one. Originators of data or our own analysis did not indicate if there is any noise. None of the three data sets has any missing values.

3.2 Intrusion detection studies on ADFA-LD dataset in literature

The ADFA-LD dataset was employed mainly for developing anomaly detection systems as reported by several studies in the literature [11, 12, 13, 14, 15]. Given that our work proposes development of a misuse intrusion detection system, it must be noted that a performance comparison between misuse and anomaly detection systems is not meaningful. Nevertheless, this section presents an overview of anomaly detection studies conducted on the ADFA-LD dataset for the sake of completeness.

Table 1
Attacks modeled in ADFA-LD dataset

Attack name	Effect	Vector
Adduser	Add new superuser	User side infected file
Hydra_FTP	Password bruteforce	FTP brute force using hydra
Hydra_SSH	Password bruteforce	SSH brute force using hydra
Java_Meterpreter	Java based meterpreter	Tiki wiki vulnerability exploit
Meterpreter	Linux meterpreter payload	User side infected file
Web_Shell	C100 Webshell	PHP remote infected file

Table 2

ADFA-LD data subsets and compositions

Data type	Type of traces	Number of traces
TRAINING_DATA_MASTER (TDM)	Normal	833
VALIDATION_DATA_MASTER (VDM)	Normal	4372
ATTACK_DATA_MASTER (ADM)	Adduser	91
	Hydra-FTP	162
	Hydra-SSH	176
	Java-Meterpreter	124
	Meterpreter	75
	Web-Shell	118

Xie et al., used a short sequence method for feature extraction and formulation on the ADFA-LD dataset and applied one-class support vector machine (SVM) for multiclass classification [11]. They reported 80% accuracy with false positive rate (FPR) of 15%. In another study, Xie et al. reduced the dimensions of the ADFA-LD dataset using a frequency-based multiclass model [12]. They applied kNN and k-Means Clustering (kMC) algorithms for detection of anomalies. Their main conclusion was that the kNN or kMC algorithms were not promising to detect attacks as their kMC implementation resulted in 60% accuracy with 20% FPR. In another study, Doyle III created frequency-based binary class model using the N-gram method and employed the kNN and SVM classifiers, which reportedly delivered 60% accuracy [13].

Haider et al. employed the zero-watermark algorithm in two different studies. In their first study, they proposed a character data zero-watermark inspired statistical based feature extraction strategy for integer data [14]. They evaluated the performance of RBF kernel, SVM and kNN classifiers on the binary class version of the anomaly detection problem. They used several metrics to report their results including the detection rate (DR), which was defined as the ratio of the number of detected anomalous activities to all the anomalous instances in the testing set, and false alarm rate (FAR) which they defined as average of false positive and negative rates, (FPR $+$ FNR)/2. They reported 78% DR with 21% FAR. In their second study, they proposed a new binary class host-based anomaly detection (HADS) model by implementing a semi-supervised approach to detect abnormal behaviors [15]. They extracted abstract hidden representation of system calls by using and expanding integer zero-watermarking algorithm. Thereafter, they applied several supervised machine learning algorithms such as naive Bayes, extreme learning machine (ELM), and support vector machine (SVM) to evaluate the performance of their generated model. They reported 100% DR and 0% FAR values on the testing data set. All anomaly detection studies except the one by Haider et al. [15] on the ADFA-LD dataset resulted in intrusion detection designs with lower-than-acceptable performance profiles.

3.3 PCA for sequences of system call traces

Principal component analysis (PCA) technique is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of linearly uncorrelated variables called principal components. The number of principal components is less than or equal to the number of original variables. This transformation is defined in such a way that the first principal component has the largest possible variance (that is, it accounts for as much of the variability in the data as possible), and each succeeding element or component, in turn, has the highest variance possible under the constraint that it is orthogonal to the preceding components. The resulting vectors form an uncorrelated orthogonal basis set. The principal components are the eigenvectors of the covariance matrix, which is symmetric. Forming these orthogonal vectors together into a matrix results in a linear space. Each original data instance vector can be represented by a smaller number of variables, which makes the PCA a useful tool for analysis in high dimensional spaces.

To apply the Eigenfaces method, which we renamed as Eigentraces, on the ADFA-LD system call trace data set we need to consider seven sets of system call traces, one set for each of seven classes ( $i=$ 1, 2, …, 7). The first step in the application of the feature extraction procedure is to set the value of window length on the trace data: we take the shortest trace length which is 75 as the basis and set the window length, $N$ , to that value plus one. Let a matrix $\bm{D}_{i}$ for class $i$ be formed from $L_{i}$ windowed system call traces, $\bm{t}_{i,j}$ , as $N\times 1$ column vectors. The value of $L_{i}$ varies and depends on the class $i$ and each trace $\bm{t}_{i,j}$ contains $N$ system calls. Then we have the following:

$\displaystyle\bm{D}_{i}=[\bm{t}_{i,1}:\bm{t}_{i,2}:\ldots:\bm{t}_{i,L_{i}}],% \text{ for }i=1,2,\ldots,7,$

where $L_{i}$ represents the count of windowed traces in class $i$ ; and $\bm{t}_{i,j}$ represents the $j$ -th trace of class $i$ and is an $N$ -dimensional column vector of system call values.

The average trace vector is calculated to normalize the trace vectors. The average trace vector $\bm{c}_{i}$ for class $i$ is an $N\times 1$ vector defined as follows where $N$ is the length of traces:

$\displaystyle\bm{c}_{i}=\frac{\bm{t}_{i,1}+\bm{t}_{i,2}+\ldots+\bm{t}_{i,L_{i}% }}{L_{i}}\text{ for }i=1,2,\ldots,7.$

The matrix $\bm{P}_{i}$ is formed and has $L_{i}$ columns, where each column is a normalized trace vector, and is computed as follows:

$\displaystyle\bm{P}_{i}=[\bm{t}_{i,1}-\bm{c}_{i}:\bm{t}_{i,2}-\bm{c}_{i}:% \ldots:\bm{t}_{i,L_{i}}-\bm{c}_{i}]\text{ for }i=1,2,\ldots,7.$

Given the matrix $\bm{P}_{i}$ , the $N\times N$ covariance matrix $\bm{Q}_{i}$ is computed as shown in the following equation:

$\displaystyle\bm{Q}_{i}=\bm{P}_{i}\bm{P}_{i}^{T}\text{ for }i=1,2,\ldots,7.$

In the next step, eigenvectors $\bm{e}_{i,j}$ , for $i=$ 1, 2, …, 7 and $j=$ 1, 2, …, $N$ , of the covariance matrix are calculated. The matrix $\bm{E}_{i}$ is defined using $N$ eigenvectors: eigenvectors $\bm{e}_{i,j}$ are stored in $\bm{E}_{i}$ as $N\times 1$ column vectors:

$\displaystyle\bm{E}_{i}=[\bm{e}_{i,1}:\bm{e}_{i,2}:\ldots:\bm{e}_{i,N}]\text{ % for }i=1,2,\ldots,7,$

where $\bm{E}_{i}$ is $N\times N$ matrix.

To generate templates for storage, compute the set of vectors $\bm{z}_{i,j}$ for each trace in class $i$ by calculating the product of $\bm{E}_{i}^{T}$ with the difference of trace $\bm{t}_{i,j}$ and $\bm{c}_{i}$ :

$\displaystyle\bm{z}_{i,j}=\bm{E}_{i}^{T}(\bm{t}_{i,j}-\bm{c}_{i})\text{ for }i% =1,2,\ldots,7\text{ and }j=1,2,\ldots,L_{i},$

where the value of $i$ refers the corresponding class and $L_{i}$ represents the number of traces (76 $\times$ 1 column vectors) in class $i$ .

To determine the class membership of a newly encountered trace, $\bm{t}_{\textit{test}}$ , subtract the average trace $\bm{c}_{i}$ from $\bm{t}_{\textit{test}}$ then compute the inner product of resultant vector with $\bm{E}_{i}^{T}$ :

$\displaystyle\bm{z}_{\textit{{test}(i)}}=\bm{E}_{i}^{T}(\bm{t}_{\textit{test}}% -\bm{c}_{i})\text{ for }i=1,2,\ldots,7.$

If only the nearest neighbor is considered for class association of the test trace, then the minimum Euclidean distance $d$ between $\bm{z}_{i,j}$ and $\bm{z}_{\textit{test}(i)}$ for a specific $i$ value indicates that the $\bm{t}_{\textit{test}}$ trace belongs to class $i$ :

$\displaystyle d=\min_{i,j}||\bm{z}_{\textit{test}(i)}-\bm{z}_{i,j}||\text{ for% }i=1,2,\ldots,7\text{ and }j=1,2,\ldots,L_{i},$

where $L_{i}$ refers to the number of traces in class $i$ .

3.4 K-nearest-neighbors classification algorithm

The k-nearest neighbor (kNN) algorithm is a nonparametric technique for classification. It is a competitive algorithm based on many studies reported for clustering, classification, pattern recognition, and categorization [16]. Training examples for kNN are feature vectors in multi-dimensional space in which each example is assigned a class label. The basic idea for classification phase behind this instance-based learning method is to calculate the distance between an unlabeled test vector and $k$ nearest neighbors where the $k$ value is provided by the user. One common and easy to compute distance metric is the standard Euclidean distance although there are many others [18, 19, 20].

The kNN classifier was applied to a variety of computer security datasets with notable success [17]. Yihua et al. used the kNN classifier on DARPA computer security dataset with different $k$ values and achieved 100% detection rate [16]. In another study on the KDD Cup’99 dataset, Swathi et al. used a modified version of kNN which he called “Fast k-nearest neighbor” [19]. Xie et al. applied kNN in conjunction with the kMC algorithm on the ADFA-LD dataset examining different $k$ values and used the Principal Component Analysis (PCA) to reduce the dimensionality of feature vectors [12]. They reported an accuracy value of 60% with 20% false positive rate.

Table 3
Number of traces in training and testing data sets

Trace type	Instances in training set	Instances in testing set
Normal	833	4372
Adduser	54	37
Hydra_FTP	97	65
Hydra_SSH	105	71
Java_Meterpreter	74	50
Meterpreter	45	30
Web_Shell	70	48
Total	1278	4673

4. Simulation study

4.1 Training and testing data

There are seven different trace files or attack types in ADFA-LD, which are distributed across three different folders which are named as TDM, VDM and ADM. To generate the training data set for misuse detection, we used the entire TDM and a 60% of the ADM. The ADM data set was split into two subsets at 60% and 40% randomly and 60% split was included in the training data set. Consequently, the training data set contains 833 normal traces and 445 attack traces. For the testing data set, we have used the entire VDM with 4372 normal traces and the remaining 40% of ADM with 301 attack traces. Table 3 presents the detailed information about how the each attack and normal instances in the ADFA-LD data set was split into training and testing data sets.

The length of traces in the ADFA-LD data set varies from 75 to 4494 system calls. To apply the Eigentraces methodology on this dataset, all traces must be equalized in length. One option is to define a window of size $w$ , which is sliding over traces by a shift count or length $s$ where $s$ is equal to or smaller than $w$ . Each $w$ system call number sequence in trace files will form a single pattern. One drawback of this method is that the length of a trace might be incompatible with the shift size for the very last window of a trace file. In other words, the shift size might be greater than the remaining trace length. In those cases, we added “0.1” as a dummy system call number as many times as needed to the end of trace. Next an illustrated example is presented. Suppose there is a sequence of numbers with length of 10 as below:

Assume a window size of 6 ( $w=$ 6) and shift length of 5 ( $s=$ 5). Consider each rolled window as an input pattern: accordingly $\alpha_{1}$ to $\alpha_{6}$ forms a pattern as shown below:

Next shifting the window to the right over this sequence by 5 positions or elements will result in the pattern as below:

A number that is outside the range of valid system call values can be used for any blank position so as not to affect the outcome. For instance, if the range of numbers for valid system calls is between 1 and 100, any value or number outside this range can be added for the position $\alpha_{11}$ .

For ADFA-LD, we assigned 76 as the window size and 10 as the shift length or size since the minimum trace size is 75. The window size is implied by the shortest trace length. The choice of a value of 10 for the shift size is a compromise to generate a manageable number of input patterns. This procedure is performed to convert the traces with different lengths into a set of vectors with 76 elements. These trace vectors are stored in the columns of a matrix associated with each of seven classes. Therefore, there are seven different matrices where each one is associated with data for one of six attack classes and the normal class. There is a separate matrix for the training data set and a second matrix for the testing data set. Table 4 shows the dimensions of training and testing data matrices for each of six attack classes and the normal class.

4.2 Eigentraces methodology

Table 4
Training and testing matrices

$i$	Class	Training Matrix ( $D_{i}$ )	Testing Matrix ( $T_{i}$ )
1	Normal	76 $\times$ 25689	76 $\times$ 185196
2	Adduser	76 $\times$ 2004	76 $\times$ 1631
3	Hydra_FTP	76 $\times$ 2592	76 $\times$ 1482
4	Hydra_SSH	76 $\times$ 4091	76 $\times$ 2069
5	Java_Meterpreter	76 $\times$ 2864	76 $\times$ 2319
6	Meterpreter	76 $\times$ 2019	76 $\times$ 1018
7	Web_Shell	76 $\times$ 2893	76 $\times$ 2175

MATLAB ${}^{\text{TM}}$ R2016a (9.0.0.341360) was used for data preprocessing and implementation of the Eigentraces methodology. Choice of a distance metric is an important consideration in the application of Eigentraces algorithm as a distance metric is needed to compute the “similarity” of two vectors. Not only accurate and meaningful measurement of the distance between any two vectors needs to be computed, the distance metric chosen needs to be feasible with respect to its computational cost. There are no less than twenty distance metrics which can be considered for the Eigentraces algorithm. Several studies reported in the literature [18, 19, 20] had done a comparative performance analysis for most of these metrics for the face recognition problem using the Eigenfaces algorithm. In the same studies, authors tested the performance of the kNN as a classifier with a large number of distance metrics. Their findings all point at Euclidean distance metric as a very effective and low computational complexity choice. This finding led to our decision of also employing the Euclidean distance for the application of the Eigentraces and the kNN classifier algorithms. Figures 1 and 2 present the application of the Eigentraces algorithm for training and testing procedures, respectively.

Figure 1.

Eigentraces methodology for training set.

Table 5

Classifier performance evaluation for binary classification on training data split

Class	TPR	FPR	TNR	FNR	Precision	F-Measure	ROC
a. Performance metrics
Normal	1.000	0.005	0.995	0.000	0.997	0.999	0.999
Abnormal	0.995	0.000	1.000	0.004	1.000	0.998	0.999
Weighted Avg.	0.998	0.003	0.997	0.003	0.998	0.998	0.999

b. Confusion matrix
a	b	$\Leftarrow$ Classified as
8496	0	a $=$ Normal
25	5416	b $=$ Attack

Table 6

Classifier performance evaluation for multiclass case on training data split

Class	TPR	FPR	TNR	FNR	Precision	F-Measure	ROC
a. Performance metrics
Normal	1.000	0.005	0.995	0.000	0.997	0.999	0.999
Adduser	0.988	0.000	1.000	0.012	1.000	0.994	0.995
Hydra_FTP	0.992	0.000	1.000	0.008	1.000	0.996	0.993
Hydra_SSH	0.999	0.000	1.000	0.001	1.000	1.000	1.000
Java_Meterpreter	0.991	0.000	1.000	0.009	1.000	0.995	0.996
Meterpreter	0.999	0.000	0.999	0.001	0.996	0.997	1.000
Web_Shell	0.994	0.000	0.999	0.006	0.996	0.995	0.999
Weighted Avg.	0.998	0.003	0.999	0.002	0998	0.998	0.999

a	b	c	d	e	f	g	$\Leftarrow$ Classified as
b. Confusion matrix
8469	0	0	0	0	0	0	a $=$ Normal
4	654	0	0	0	3	1	b $=$ Adduser
6	0	860	0	0	0	1	c $=$ Hydra_FTP
0	0	0	1285	0	0	1	d $=$ Hydra_SSH
8	0	0	0	956	0	1	e $=$ Java_Meterpreter
1	0	0	0	0	683	0	f $=$ Meterpreter
6	0	0	0	0	0	971	g $=$ Web_Shell

4.3 Classifier model development

For classifier development, class instances in the training data set are further split into two parts at a ratio of 67% to 33%. The larger split is used to train the classifier and the smaller split is used to validate its performance. The kNN algorithm implemented in Weka Version 3.8 with $k=$ 5, which is empirically determined through an exploratory search, is trained as a classifier. Performance of the classifier is evaluated in terms of standard metrics including TPR, FPR, TNR, FNR, Precision, F-Measure and ROC for both multi and binary class problems. Values of the other parameters for the kNN classifier are as follows: distance metric is Euclidean; neighbors are weighted by the inverse of their distance; the nearest neighbor search algorithm is LinearNNSearch (by its Weka label); and all other parameters including minimization of mean squared error rather than mean absolute error (default value is “No”) and the window size which determines the maximum number of training instances maintained (default value is “No window”) are left at their default values.

Table 7
Classifier performance for binary class on test data

Class	TPR	FPR	TNR	FNR	Precision	F-Measure	ROC
a. Performance metrics
Normal	1.000	0.002	0.998	0.000	1.000	1.000	1.000
Abnormal	0.998	0.000	1.000	0.002	1.000	0.999	0.999
Weighted Avg.	0.999	0.002	0.999	0.001	1.000	0.999	0.999

b. Confusion matrix
a	b	$\Leftarrow$ Classified as
185196	0	a $=$ Normal
23	10671	b $=$ Attack

Figure 2.

Eigentraces methodology for testing set.

Table 8

Classifier performance for multiclass on test data

Class	TPR	FPR	TNR	FNR	Precision	F-Measure	ROC
a. Performance metrics
Normal	1.000	0.002	0.998	0.000	1.000	1.000	1.000
Adduser	0.992	0.000	1.000	0.008	1.000	0.996	0.996
Hydra_FTP	0.994	0.000	1.000	0.006	1.000	0.997	0.998
Hydra_SSH	0.999	0.000	1.000	0.001	1.000	1.000	1.000
Java_Meterpreter	0.994	0.000	1.000	0.005	1.000	0.997	0.999
Meterpreter	1.000	0.000	0.999	0.000	0.989	0.995	1.000
Web_Shell	1.000	0.000	1.000	0.000	1.000	1.000	1.000
Weighted Avg.	0.999	0.002	0.999	0.001	0.999	0.999	0.999

a	b	c	d	e	f	g	$\Leftarrow$ Classified as
b. Confusion matrix
185196	0	0	0	0	0	0	a $=$ Normal
6	1613	0	0	0	12	0	b $=$ Adduser
4	0	1476	0	0	2	0	c $=$ Hydra_FTP
2	0	0	2067	0	0	0	d $=$ Hydra_SSH
14	0	0	0	2301	2	0	e $=$ Java_Meterpreter
0	0	0	0	0	1018	0	f $=$ Meterpreter
0	0	0	0	0	0	2175	g $=$ Web_Shell

Two labels, namely “Normal” and “Attack”, are used for the binary classification. The multiclass problems consists of the “Normal” label and six attack labels. Table 5a and b present the performance metric values and confusion matrix for the kNN classifier applied to the binary class problem, respectively. In Table 5a, the TPR for normal class is 1.0, which means all normal traces are detected. The TPR for the attack class is 0.995, which indicates that 5416 of 5441 attack traces are detected: only 25 traces among 13937 normal and attack traces are misclassified. The overall accuracy of this model is 99.82%.

The results for the multiclass case also indicate that the kNN delivers nearly perfect prediction for each of the seven classes. As shown in Table 6a, the TPR for all seven classes is 0.988 or higher, which means 99.8% of all traces are classified correctly (Table 6b). The accuracy of this model is 99.78%.

4.4 Classifier performance

Tables 7 and 8 show the performance evaluation results for binary and multi class models, respectively,on the testing data. In those tables, part a represents the performance evaluation results and part b shows the confusion matrix for the same model.

Data in both tables indicate highly accurate results in predicting traces in term of their class memberships. For the binary class problem, 23 out of 195890 traces are misclassified. All of the 185196 normal traces and 10671 out of 10694 attack traces are correctly classified. For the multiclass model, 42 traces are misclassified. In this model, referring to Table 8, all of the 185196 normal traces, 1018 meterpreter attacks, and 2175 web-shell attacks are classified correctly. The number of misclassified traces for Adduser, Hydra_FTP, Hydra_SSH, and Java_Meterpreter are 18, 6, 2, and 16 respectively, which are less than 0.05% of total traces for each class. Furthermore 26 attack traces are misclassified as “Normal” while 16 attack traces are classified as belonging to another attack class.

In overall, the accuracy for both models (binary and multiclass classifiers) is 99.98% or higher. This suggests that the Eigentraces in combination with kNN is a very promising classification algorithm for misuse detection in operating system call traces for the ADFA-LD context.

5. Conclusions

This paper presented the development of a host-based misuse intrusion detection (IDS) system. The proposed design employs the Eigentraces methodology for feature extraction and pattern formation, and the kNN algorithm for classification on the ADFA-LD data set. The ADFA-LD dataset entails seven different classes, one for normal and six for different attacks, where data are sequences of system calls (traces) collected on the Linux operating system. The attacks include Adduser, Hydra_FTP, Hydra_SSH, Java_Meterpreter, Meterpreter, and Web_Shell. The IDS performance was evaluated for two different scenarios. In one case, two classes were created, one for the normal traces and a second class for all six attacks combined. The original ADFA-LD dataset was split into two subsets, one for training and another one for testing, where all attacks as well as the normal class patterns were included in both the training and testing datasets. The second scenario entailed addressing the problem as a seven-class case. Both the training and testing data had samples from all seven classes.

Performance of the IDS was evaluated through a simulation study. For the case of binary classification, only 23 attack traces were misclassified as normal while all 185196 normal traces and the remaining 10671 attack traces were correctly classified with no false alarms. For the multiclass case, all 185196 normal traces were correctly classified. In addition, 1018 Java_meterpreter attacks, and 2175 web-shell attacks are classified correctly. Misclassifications occurred for a small fraction of all traces for the other four attacks. 18 Adduser, 6 Hydra_FTP, 2 Hydra_SSH, and 16 Java_Meterpreter attack traces were misclassified while misclassified traces constitute less than 0.05% of total traces for each attack class. In conclusion, the proposed IDS design performance was very high for identifying attacks and predicting the type of attack.

References

Liao

H.-J.

Lin

C.-H.R.

Lin

Y.-C.

and Tung

K.-Y.

, Intrusion detection system: A comprehensive review, Journal of Network and Computer Applications 36(1) (2013), 16–24.

Scarfone

and Mell

, Guide to intrusion detection and prevention systems (idps), NIST Special Publication 800(2007) (2007), 94.

Abdi

and Williams

L.J.

, Principal component analysis, Wiley Interdisciplinary Reviews: Computational Statistics 2(4) (2010), 433–459.

Serpen

Iyer

Elsamaloty

and Parsai

, Automated lung outline reconstruction in ventilation-perfusion scans using principal component analysis techniques, Computers in Biology and Medicine 33(2) (2003), 119–142.

Turk

, Over twenty years of eigenfaces, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 9(1s) (2013), 45.

Agarwal

Jain

Kumar

M.M.

and Agrawal

, Face recognition using eigen faces and artificial neural network, International Journal of Computer Theory and Engineering 2(4) (2010), 624.

Kshirsagar

Baviskar

and Gaikwad

, Face recognition using Eigenfaces, in: Computer Research and Development (ICCRD), 2011 3rd International Conference on, Vol. 2, 2011, pp. 302–306: IEEE.

Ravi

and Nayeem

, A study on face recognition technique based on eigenface||, International Journal of Applied Information Systems 5(4) (2013), 57–62.

, The ADFA Intrusion Detection Datasets [Online]. Available: https://www.unsw.adfa.edu.au/australian-centre-for-cyber-security/cybersecurity/ADFA-IDS-Datasets/.

10.

Gideon Creech

J.H.

, Generation of a new IDS test dataset: Time to retire the KDD collection, in: 2013 IEEE Wireless Communications and Networking Conference (WCNC), 2013, pp. 4487–4492.

11.

Xie

and Slay

, Evaluating host-based anomaly detection systems: Application of the one-class SVM algorithm to ADFA-LD, in: Fuzzy Systems and Knowledge Discovery (FSKD), 2014 11th International Conference on, 2014, pp. 978–982: IEEE.

12.

Xie

and Chang

, Evaluating host-based anomaly detection systems: Application of the frequency-based algorithms to adfa-ld, in: International Conference on Network and System Security, 2014, pp. 542–549: Springer.

13.

Doyle III

W.J.

, Classifying System Call Traces using Anomalous Detection, Computer Science, 2015.

14.

Haider

and Xie

, Towards reliable data feature retrieval and decision engine in host-based anomaly detection systems, in: Industrial Electronics and Applications (ICIEA), 2015 IEEE 10th Conference on, 2015, pp. 513–517: IEEE.

15.

Haider

and Xie

, Integer Data Zero-Watermark Assisted System Calls Abstraction and Normalization for Host Based Anomaly Detection Systems, in: Cyber Security and Cloud Computing (CSCloud), 2015 IEEE 2nd International Conference on, 2015, pp. 349–355.

16.

Yihua Liao

V.R.V.

, Use of K-Nearest Neighbor classifier for intrusion detection, Computers & Security 21(5) (2002), 439–448.

17.

Swathi

D.S.L.K.

, Network Intrusion Detection Using Fast k-Nearest Neighbor Classifier, NSCS, 2014.

18.

Greche

Jazouli

Es-Sbai

Majda

and Zarghili

, Comparison between Euclidean and Manhattan distance measure for facial expressions classification, in: Wireless Technologies, Embedded and Intelligent Systems (WITS), 2017 International Conference on, 2017, pp. 1–4: IEEE.

19.

Singh

and Pandey

, An euclidean distance based KNN computational method for assessing degree of liver damage, in: Inventive Computation Technologies (ICICT), International Conference on, Vol. 1, 2016, pp. 1–4: IEEE.

20.

Satonkar

Kurhe

and Khanale

P.B.

, Face recognition using different distance measures techniques, IJIPA 3(1) (2012), 29–36.

21.

Bhavesh Borisaniya

D.P.

, Evaluation of Modified Vector Space Representation Using ADFA-LD and ADFA-WD Datasets, Information Security, 2015.

22.

Depren

Topallar

Anarim

and Ciliz

M.K.

, An intelligent intrusion detection system (IDS) for anomaly and misuse detection in computer networks, Expert Systems with Applications 29(4) (2005), 713–722.

23.

Assem

Rachidi

and Graini

M.T.E.

, Intrusion detection using bayesian classifier for arbitrarily long system call sequences, IADIS International Journal on Computer Science and Information Systems 9(1) (2014), 71–81.

24.

Kang

D.-K.

Fuller

and Honavar

, Learning classifiers for misuse and anomaly detection using a bag of system calls representation, in: Information Assurance Workshop, 2005. IAW’05. Proceedings from the Sixth Annual IEEE SMC, 2005, pp. 118–125: IEEE.

25.

Laskov

Düssel

Schäfer

and Rieck

, Learning intrusion detection: supervised or unsupervised? in: International Conference on Image Analysis and Processing, 2005, pp. 50–57: Springer.

26.

Kuchimanchi

G.K.

Phoha

V.V.

Balagani

K.S.

and Gaddam

S.R.

, Dimension reduction using feature extraction methods for Real-time misuse detection systems, in: Information Assurance Workshop, 2004. Proceedings from the Fifth Annual IEEE SMC, 2004, pp. 195–202: IEEE.