DeepReturn: A deep neural network can learn how to detect previously-unseen ROP payloads without using any heuristics

2.1. ROP attacks

Despite years of effort to mitigate the threat, ROP attack remains a widely used method to launch exploits [35,46,47,67]. In a typical ROP attack, an attacker carefully crafts a gadget chain and embeds the addresses of the gadgets in a network packet. This packet exploits a vulnerability in the server program and causes the gadget to be executed one by one. Figure 1 shows the relationship of the network packet payload, the overflowed stack, and the program’s memory address space layout. In fact, this comes from the blind ROP attack [29], which exploits a stack buffer overflow vulnerability in Nginx 1.4.0. As we can see, the addresses of gadgets (e.g., 0x804c51c, 0x804c69a) are embedded in the network packet. This relationship is an intrinsic characteristic of ROP attacks and it motivates us to search for addresses of potential gadgets in input data and then disassemble the corresponding instructions.

On the other hand, since only the address of ROP gadgets are contained in the ROP payloads, we should not directly label ROP payloads and benign data and then train a classifier on the two – because the signal-to-noise ratio is too low and it is difficult to train an accurate classifier.

OPEN IN VIEWER

Fig. 1.

Relationship between network packet payload, stack layout, and address space layout.

In this subsection, we classify the existing detection methods against ROP attacks into six categories and explain why they fail to meet the four requirements set in Section 1.

Heuristic-based detection methods look for abnormal execution patterns. DROP [10] checks whether the frequency of executed return instructions exceeds a certain threshold. kBouncer [38] and ROPecker [11] detect abnormal pattern of indirect branches. Although these methods can detect conventional ROP attacks, they leverage certain heuristic-based thresholds as critical detection criteria. Göktaş et al. point out in [21] that they can be bypassed by carefully constructed gadget chains which violate the defender’s assumptions.

Control Flow Integrity (CFI) [1,4,14,65,68] involves two steps, i.e., detection and response. CFI detects whether any control flow transfer violates the pre-defined control flow graph (CFG). If so, it terminates the program. Theoretically, CFI can detect all ROP attacks since they inevitably violate the CFG. However, due to the difficulty of point-to analysis, obtaining a perfect CFG is difficult (if not impossible) [7]. In practice, coarse-grained CFI implementations adopt an approximated CFG, which is not as secure as it should be. Consequently, attackers can leverage the extra edges in the CFG to bypass them [8,14,15,36]. On the other hand, fine-grained CFI provides a strong security guarantee at the cost of significantly high runtime overhead; e.g., 19% in [39] and 52% in [31], which is impractical. Meanwhile, CFI requires either instrumentation to the program binary or modifications to the compiler.

There are also signature-based detection methods against ROP, i.e., n-ROPdetector [63]. n-ROPdetector checks whether a set of addresses of API functions appear in network traffic. While this method can detect some ROP attacks, the address of APIs can be masqueraded to evade the detection. For example, an attacker can represent an address as the sum of two integers, and calculate the address at runtime.

Speculative-code-execution-based detection (ROPscan [41]) searches network traffic or files for any code pointers and starts speculative code execution from the corresponding instruction. If four or more consecutive gadget chains can be successfully executed, ROPscan considers the input data as a ROP exploit. Note the threshold four is empirically determined in the experiment; however, check my profile [59] finds that benign data can produce a gadget chain that has up to six gadgets. Meanwhile, real gadget chains need not to be very long to be useful. Thus the selection of the threshold is challenging. In fact, this also motivates the use of deep neural network as a classifier because no such thresholds need to be provided.

Check my profile [59] statically analyzes the data region and looks for any code pointers that form potential gadget chains. To avoid false positives, it checks whether the gadget chain eventually calls an API function – which most useful exploits will do. Unfortunately, due to the nature of static analysis, the detection can be bypassed when the address of API function is masqueraded, e.g., as a sum of two integers.

Statistical-based detection methods [18,40] first extract certain meta-information (e.g., micro-architectural events) about the program execution and then train a statistical machine learning model (e.g., SVM) to distinguish ordinary execution from ROP gadget chain execution. They can detect various types of ROP attacks. However, they require instrumentation to acquire such information. Moreover, they use relatively small training dataset, which leaves plenty of spaces for improvement. For example, EigenROP [18] achieves 80% detection rate and 0.8% false positive rate, whereas we get 99.3% detection rate and 0.01% false positive rate. Note our false positive rate means 0.01% of the potential gadget chains, and the number of such chains is significantly smaller the number of program inputs (see Section 7), so our accuracy is much better than EigenROP.

4.1. Disassembly of individual addresses

We first create a memory dump of the protected program. The memory dump contains the addresses and contents of all memory pages that are marked as executable. Oftentimes, these pages are mapped from the text segment of the protected program or any loaded shared libraries (e.g., libc.so, etc). The ROP gadgets must fall inside the memory dump, otherwise it is not executable and the attack will fail.

Then we consider every four bytes (on 32-bit system)1

We start disassembling from any identified addresses using Capstone [43]. The disassembling can stop in two ways: (1). An invalid or privileged instruction is encountered or the disassembly reaches the end of code segment. In this case, the current address is ignored in subsequent analysis. (2). An indirect branch, i.e., ret, jmp, or call is encountered. Then the instruction sequence (from the instruction at the starting address all the way to the indirect branch) is considered a gadget-like instruction sequence.

Having obtained the set of gadget-like instruction sequences, we need to figure out how they could be chained together, in a similar way as an attacker chains gadgets into a gadget chain. Specifically, if we find an address at offset n in the data which points to a gadget-like instruction sequence, we check if any one of the next ten addresses, i.e., address at n + 4 , n + 8 , n + 12 ,…, n + 40 in the data also points to a gadget-like instruction sequence. If so, we “chain” it together with the previous gadget-like instruction sequence and repeat the process. Otherwise, we end the “chaining” process.

For any “chain” that has at least two addresses, we collect the corresponding gadget-like instruction sequences and concatenate them into a gadget-chain-like instruction sequence.

Note the maximum ten is determined by the observation that most gadgets in our dataset only pop less than five integers from the stack to the registers, and all of them pop less than ten integers; so ten is sufficient to capture the next address in a “chain”.

In other words, a ROP attack cannot spread the addresses of gadget chains in the payload arbitrarily; otherwise, even if the control flow is successfully hijacked, the subsequent gadgets will not execute one by one – because upon return from the previous gadget, the address of the next address is not on the top of the stack.

When we look for the next “chain”, we skip the addresses and the corresponding instruction sequences that are already part of a “gadget chain”. For example, if a “chain” contains five “gadgets”, next we start from the sixth address and repeat the “chaining” process. We repeat the process on every address to collect all possible gadget-chain-like instruction sequence. In this way, we get the first training dataset.

To efficiently implement the above algorithm, we start multiple parallel threads to analyze different addresses. Moreover, if an address is already examined and found to be pointing to a gadget-like instruction sequence, we cache it in a global table. In this way, the disassembly and analysis is not repeated on the same address. Besides, we also process multiple inputs simultaneously to utilize all CPU cores.

The reader may take Fig. 1 as an example of the ASL-guided disassembly during the production phase. Suppose we start from offset n of the data, we first encounter a candidate address 0x456af094. We find this address is not inside the memory dump, which indicates it is not mapped or not marked as executable in the protected program, so we move on to the next 4 bytes starting at offset n+0x04. The next two addresses are 0x41414141 and they both do not lead to a potential gadget chain. Next, we move to offset n+0x0c and process 0x0804c69a. Note this address is inside the memory dump and we start disassembling from it. The result is a potential gadget with three instructions: pop esi; pop edi; ret;. We continue the process and then identify two other addresses (0x080bcbec, 0x0804c51c) and their corresponding instructions. Eventually, we end up with a potential gadget chain with three gadgets (pop esi; pop edi; ret; pop eax; ret; pop esi; pop edi; pop ebp; ret;). In the training phase, however, when we collect gadget-chain-like-instruction sequences, we process benign input data using the same approach.

It is noteworthy that we start ASL-guided disassembly from EVERY byte of the input data. This is because we are dealing with data, so code or memory alignment actually does not apply, and any four-bytes can be an address. In fact, in Fig. 1, we also treat the four-byte data at n+1 (0x6af09441), n+2(0xf0944141) and n+3(0x94414141) as addresses and start ASL-guided disassembly. Though they do not lead to gadget-like instruction sequences and are ignored in the subsequent analysis.

6.1. Data representation and preprocessing

In order to make our two datasets (i.e., the gadget-chain-like instruction sequences and the real gadget chains) tractable for the neural network, we need to convert every instruction in them into numerical data. Recall that instructions are binary data by nature. We first convert every byte in an instruction sequence into its hex value (0-255). For example, the instruction sequence “mov eax, 0x1; pop edx; ret” is converted to “[0xb8, 0x01, 0x00, 0x00, 0x00, 0x5a, 0xc3]”.

We can simply scale these values to 0-1 by a division of 255 and then input them into the neural network. However, this is inappropriate for our data. Since neural networks multiply the input data with certain weight parameters, such representation leads to an implicit relative order between different byte values. For example, in the above example, the neural network implicitly assumes the “ret” (0xc3) is larger than “pop edx” (0x5a), which is meaningless regarding instructions.

To address this problem, we instead use one-hot encoding. We represent each byte value by a 256 × 1 vector, with all but one positions to be zero, and the position that corresponds to the byte value to be one. For example, the “ret” (0xc3, 195) will be represented by { 0 , … , 0 , 1 , 0 , … , 0 } , where we have 195 zeros in front of the one, and 60 zeros behind the one.

In other words, an instruction sequence that has n bytes is represented by X = { X 1 → , X 2 → , … , X n → } , where X i → is a 256 × 1 one-hot vector. Alternatively, one can view the input as a n by 256 matrix.

Since instruction sequences usually have different lengths, padding is applied to make them have the same length. We first find the longest instruction sequence (in bytes). For shorter ones, we append the one-byte nop (0x90) instruction at the end of them until they all reach the same length.

An alternative way to preprocess the data is to do word embedding [27], which is widely used in Natural Language Processing (NLP). Word embedding maps every word in the data to a fixed length vector. In the evaluation section, we show that embedding, compared to one-hot encoding, provides similar accuracy while requires longer training time. Consequently, one-hot encoding is more suitable for our task.

After the data preprocessing, we use a customized neural network to classify whether a potential gadget chain is benign or real. The architecture of our neural network (shown in Fig. 4) is a convolutional neural network (CNN), which is a feed-forward neural network that is good at learning data with spatial structures and capturing their local dependencies. Typically, a CNN has one input layer, multiple convolution layers, multiple activation layers, and one output layer. In particular, all of the convolution layer in the network is 1D convolution layer which is shown in Fig. 3.

The 1D convolution layer involves a mathematical operation called convolution. Suppose X represents the one-hot encoded instruction bytes, w represents the convolution kernel (weight vector) whose length is m. Then the convolution between X and w is a matrix X ∗ w , whose ith column can be written as: (1) ( X ∗ w ) i = ∑ j = 1 m X i − j + m / 2 · w j

The convolution aggregates information from adjacent bytes. This information includes certain byte values, the ordering of bytes, etc. The convolution layer is followed by a nonlinear activation layer (e.g., ReLU) to denote whether certain information is present or not. We stack three layers of convolution and activation to gradually capture higher level of information of the input bytes, e.g. the presence of a certain gadget, the ordering of different gadgets, the repetition of certain patterns, etc. The higher level of information is more abstract and difficult to extract or represent (similar to the case in image classification), but it is more related to the classification task, i.e., whether the chain is benign or real. The last activation layer is followed by a fully connected layer and another activation layer output to give a classification output, benign (0) or real (1).

Since the input X is fixed, only the weights w influence the output. These values are not determined via heuristics or expert knowledge, as in many previous ROP detection methods. Instead, they are trained (recursively updated) by minimizing the differences between the true labels and the network’s outputs. For details about CNN, please refer to [22,26].

OPEN IN VIEWER

Fig. 3.

Data representation and the first 1D convolution layer.

OPEN IN VIEWER

Fig. 4.

Architecture of the CNN used in DeepReturn.

The architecture of our neural network is illustrated in Fig. 4, which is a zoom-in version of the CNN in Fig. 2. We first use a convolutional layer with 64 convolution filters with the kernel size of 7 (length of the convolution filter is 7). Then we perform batch normalization (BN) before a nonlinear activation function, which in this case is a rectified linear unit (ReLU). ReLU is a simple rectifier with the form of f ( x ) = max ( x , 0 ) . After the ReLU activation, we apply a 50% dropout to prevent overfitting. Then we repeat this Convolution-BN-ReLU-Dropout structure for two more times. In particular, we use 32 convolutional kernels with the size of 5 and 16 convolutional kernels with size 3 for the next two layers. Then the output from the last Convolution-BN-ReLU-Dropout structure is feed into a fully connected layer. Finally, we use the softmax activation function as a classifier. It is worth mentioning that we do not include pooling layers which are widely used in image classification tasks, because our entire input is meaningful and downsampling our input vector would yield a completely different gadget chain.

We use a grid search [24] to fine-tune the above configurable hyper-parameters (e.g., dropout rate, the filter sizes, etc) in our model and find the best set of values. Details are provided in Section 7.

This network includes some modern techniques in deep learning such as BN [25] and dropout [58] to improve the performance. BN is a method to reduce internal covariance shift by normalizing the layer inputs in the neural network [25], and has been shown to improve learning. Dropout is a simple way to prevent a neural network from overfitting by randomly dropping a percentage of neurons during the training phase. These methods have been widely used in recent years and seen great successes.

We train the CNN after collecting training samples (gadget-chain-like instruction sequences and real gadget chains) for a specific program. We use an optimization procedure called stochastic gradient descent (SGD) with learning rate 0.1 and momentum [42]. In particular, we input a mini-batch of the training samples into the network, then compute the outputs, the errors and the gradients for the input. The gradients are calculated by backpropagation to determine how the network should update its internal weights; e.g., w 1 , w 2 and w 3 in Fig. 3, which are used to compute the representation in each layer from the representation in the previous layer. We repeat this process for many mini-batches until the errors converge or the maximum training epoch is reached. In practice, we also test another optimizer, i.e., Adam, and both the accuracy and the convergence speed are similar to those of the SGD, so we choose to use SGD as our optimizer.

Note we set forth a goal (R2) to reach close to zero false positive rate for the DeepReturn. This is motivated by the fact that false positives can really irritate the system admin and undermine the usefulness of the detection method, or even impact its security. To reduce the false positive rate, we use a common technique, i.e., penalizing false positives more than false negatives. Specifically, we use the notation of “penalizing factor” to describe how much more a false positive contributes to the loss function compared to a false negative. A penalizing factor 1 means no difference between a false positive and a false negative, while a penalizing factor of 5 means a false positive contributes to the loss 5 times more than a false negative does. In the evaluation section, we empirically decide a value to achieve a very low false positive rate and a still high detection rate.

7.1. How is big data used to generate benign training dataset?

Collecting abundant training data is one of the most challenging tasks when utilizing the neural network. We use the following three input datasets to generate the gadget-chain-like instruction sequences: (A) a medium size HTTP traffic dataset in [64]; (B) all images in ImageNet [16] (1.2 TB in total); (C) all PDF files from arXiv [20] (800 GB in total). We use all (A)(B)(C) for the two web servers, i.e., nginx and apache, to do ASL-guided disassembly (described in Section 4) and generate gadget-chain-like instruction sequences. Similarly, we use all (B) for the image processor (i.e., ImageMagick), and all (B)(C) for the two ftp servers (i.e., proftpd and vsftpd) to do the same job. As a summary, we consider all valid inputs for the target program. For example, all three datasets all typical traffic that pass web servers; image processor, however, only deals with images.

To quickly process the huge amount of data, we use a Google Cloud Platform instance with 96 vCPUs. The memory dump size, input data, generation time, and the number of generated gadget-chain-like instruction sequences are shown in Table 2. As a brief summary, we generate 26k to 105k gadget-chain-like instruction sequences for different programs, respectively.

It can be verified here that we do need a classifier in DeepReturn. Otherwise, if we simply do ASL-guided disassembly on input data and treat all potential gadget chains as real gadget chains, all gadget-chain-like instruction sequences generated above will become false positives. Once a deep learning model is trained for a program we intend to protect, its deployment will assume a representative server setup in practice. However, it should be noticed that the model training stage is a separate offline machine learning process, where the training data is prepared in advance. Since big data is usually involved in preparing the training data, we assume that some enterprises do not have the needed computing resources, and hence view a public cloud provider (i.e. Google Cloud) as a viable option.

As explained in Section 5, for each program, we generate the same number of real gadget chains as gadget-chain-like instruction sequences. The generation is rapid and is done on a local workstation. Now that we have five datasets for the five programs, respectively. Each of them contains two smaller datasets: the gadget-chain-like instruction sequences and the real gadgets chains, which have the same number of samples.

Hyper-parameter tuning is a very important aspect of deep learning applications. Different hyper-parameter sets can lead to very different results. In this subsection, we use grid search [24] to empirically find the best hyper-parameters for our CNN. Grid search exhaustively searches the hyper-parameter space to look for the combination of hyper-parameters that yields the best result. In DeepReturn, we have a handful of hyper-parameters and the grid search works well.

We implement our model using Keras [12] and run it on NVIDIA Tesla P100 GPU. We ran the experiment under each setting for three times and take the average value of the accuracy. This averaged accuracy is the primary factor to compare different hyper-parameter settings. However, simply rely on accuracy leads to a problem. For example, let us assume 128 convolution filters already leads to the best result. Doubling the number and make it 256 is likely to be a waste of training time – but the accuracy might see a tiny increase, say 0.05%. So we also consider how fast the hyper-parameters lead to the convergence of error rate, measured in training epochs. If hyper-parameter set a is less than 0.05% better than set b in accuracy, but at the same time the training epochs is 20% more, we favor set b. In this way, we can not only get high accuracy, but also avoid unnecessary training time.

We use the nginx dataset to fine-tune the hyper-parameters. We split the whole dataset into two subsets: a training dataset that is 80% of the whole dataset (80% of the samples from both labels), selected at random; a test dataset that is the rest 20% of the samples. We use the training dataset to train the model and fine-tune the hyper-parameters.

The parameters we aim to tune and their corresponding candidate values are listed in Table 3. After a grid search, we find the best set of hyper-parameters, which is highlighted in Table 3 (these are also the values included in Section 6). We achieve 99.6% accuracy under this best setting.

We also consider using word embedding to preprocess the data, which works very well in NLP. But it does not help in our task. It produces similar accuracy to one-hot encoding but the required training time is 30% longer. The longer training time can be explained by the fact the embedding layer itself needs to be trained along with the deep neural network. Also, instruction sequences are inherently different from language data, so the embedding is unable to improve the result.

We do not use grid search to find the best penalizing factor (introduced in Section 6.2). In fact, we not only want to minimize the false positive rate, we also want to maintain a high accuracy. However, using penalizing factor to decrease false positive rate inevitably increase false negative rate and may impact the overall accuracy, so a decision on the balance is better made manually. To illustrate this, we test an extremely large penalizing factor, 100, and the false positive rate is 0. However, the false negative rate is high and the overall accuracy drops to 92%, which is undesirable. On the other hand, if we use a trivial penalizing factor 1, the false positive is 0.1%. After manually testing several penalizing factors, we find a factor of 5 to be a good balance, where the false positive rate is 0.01% and the detection rate is 99.3%. Since we have same amount of samples for the two labels, so accuracy = ( detection rate + ( 1 − false positive rate ) ) / 2 , thus the overall accuracy is still 99.6%.

A quick estimation demonstrates what 0.01% false positive rate means in real world. Suppose a web server receives 10 TB incoming traffic per day, since the number of potential gadget chains is rather small (40, 674 out of 2 TB benign data), it is estimated only 20 false positives will be reported in 24 hours. Note the 10 TB incoming traffic is equivalent to 1 Gbps all the time during 24 hours, which is not a trivial amount, so the estimation is not under-estimating.

In this subsection, we evaluate the accuracy of DeepReturn against the five commonly used programs.

For every program, we select 80 % of the dataset as training data the rest 20% as test data. We use the best hyper-parameters from the previous subsection, make 5-fold cross validation and the evaluation result is shown in Table 4. We reach 99.3% detection rate and 0.01% false positive rate for nginx 1.4.0, 98.7% detection rate and 0.04% false positive rate for apache 2.4.18, 98.3% detection rate and 0.05% false positive rate for proftpd 1.3.0a, 99.5% detection rate and 0.02% false positive rate for vsftpd 3.03, and 98.1% detection rate and 0.02% false positive rate for ImageMagick 7.08, respectively. As we can see from the results, DeepReturn has very high detection rates and very low false positive rates, and work well on different programs.

The high accuracy shows that roughly 50k-100K samples are sufficient to train an accurate classifier for potential gadget chains. In contrast, for image classification, millions of samples are required to train neural networks [16].

Besides CNN, there are also other candidate classifiers, e.g., RNN (LSTM), MLP and SVM. Our evaluation shows that the CNN is best suitable for this particular task. Due to space limitation, detailed comparisons are shown in the Appendix.

It is also noteworthy that the best hyper-parameters tuned on nginx dataset work well across different programs. This indicates although the specific instruction sequences are different in different programs (thus a dedicated CNN is needed), it has something in common across different programs.

We observe that the training time roughly increases linearly proportional to the size of the training data. However, once the neural network is trained, the test speed is fast and less sensitive to the amount of test data. In the first row, it only takes 0.6 seconds to classify all test data for nginx (20% of all samples, 16,270 in total).

We also test what would happen if the amount of training data is small. In particular, we train the same model with 5% of the nginx data (4K samples). We achieve 95.1% detection rate and 1.2% false positive rate. They are considerably inferior to the best achievable result. It validates that deep neural networks work better with larger training data since the networks can discover intricate features in large data sets instead of over-fitting small training data and missing key features.

In this subsection, we test DeepReturn against real-world ROP exploits that are collected in-the-wild or generated by ROP exploit generation tools (i.e., Ropper [49] and ROPC [37]).

The motivations behind these experiments are two-fold. Firstly, in the previous subsection, we show that the trained deep neural network of DeepReturn is an accurate classifier with a high detection rate and low false positive rate. However, despite our efforts to make the real gadget chains valid, they are not necessarily useful from an attacker’s point of view. Therefore, we need a more direct evaluation here to show DeepReturn is able to detect real-world ROP exploits (that are not part of the training data). Secondly, since the real gadget chains are directly generated (not from ASL-guided disassembly), we also need to show the ASL-guided disassembly is capable of correctly identify the addresses of gadgets in a real gadget chain, which is the basis for our approach.

Among the five tested programs, nginx and proftpd have known vulnerabilities and we directly exploit them with ROP attack. For the rest three, we use the latest version so there is no known vulnerability. Instead, we inject a trivial stack buffer overflow vulnerability into each of them to make the exploiting possible. In fact, as long as the exploit is a ROP attack, the underlying detail of the vulnerability has nothing to do with the effectiveness of our method, so the injected vulnerabilities do not undermine our evaluation.

For each vulnerable program, we first obtain one ROP exploit that leads to a shell. For nginx, we use the attack script published in BROP [3]. For proftpd, we use the attack script published in Exploit-DB2

To further create test samples, we use Ropper [49] to generate ROP exploits that execute mprotect or execve. Ropper can generate different exploits because we can block the used gadgets and force it to generate new ones. Meanwhile, we create ROP exploits with ROPC [37], which is a ROP compiler that can compile scripts written in a special type of language (ROP Language, ROPL) into a gadget chain. Although in Section 5 we are unable to use these tools to generate a large number of real gadget chains, it successfully generate sufficient amount of samples for us to test DeepReturn.

To sum up, for each of the five vulnerable programs, we collect and create 20 ROP exploits. All of them are manually verified to be working. i.e., achieving their designed functionality, for example getting a shell or executing mprotect.

We then test DeepReturn against all of the 100 exploits. We first observe that the ASL-guided disassembly successfully extracts the gadget addresses embedded in the payload and obtains the corresponding instruction sequences. Subsequently, the DNN correctly classifies all of the potential gadget chains as real gadget chains. This demonstrates the ASL-guided disassembler and the neural network synergize well and the system works as designed. DeepReturn is able to detect real-world ROP exploits.

Thanks to the generalization capacity of DNN, the detection capacity of DeepReturn is not restricted to the ROP attacks it sees during training. In fact, it detects the blind ROP attack which is not part of the training data. Furthermore, ROP exploits generated by ROPC can have more than 100 gadgets, which are three times longer than the longest one in the training dataset. They are also correctly detected.

We conduct an experiment to test whether the technique of transfer learning could be adopted by DeepReturn. Essentially, we prepare two datasets using two different programs: proftpd and vsftpd, following the methods described in Section 4 and 5. Then a model is trained using the proftpd dataset from scratch, which is called baseline model. After the baseline model is trained, another model is trained using the vsftpd dataset, but the weights are initialized using the weights in the already-trained baseline model. We keep the hyper-parameters of the baseline model to be consistent with the ones in other experiments.

We first investigate whether a baseline model trained by the proftpd dataset could be used to detect ROP payloads against vsftpd. The accuracy of such investigation is 89.63% with a false positive rate of 2.94%. On one hand, the outcome implies that directly using a model trained for program A to detect ROP payloads against program B would result in unacceptable accuracy. On the other hand, since 89.63% is substantially higher than what is expected by us, the outcome implies that the “knowledge” learned to detect ROP payloads against proftpd might be transferable when a model is learned to detect ROP payloads against vsftpd.

Then we initialize a second model borrowing the weights from the baseline model, and use the vsftpd dataset to train the second model. The performance stop improving after 4.2 minutes training (5 epochs). Note that the baseline model here can be trained until it converges in 13 minutes. Therefore, the time for transfer learning instead of training from the scratch is 67.69% less. The accuracy stops improving at 97.77%, and the false positive rate drops to 0.04%. When compared to the vsftpd results shown in Table 4, we found that 97.77% is slightly lower than 99.5%, and 0.04% is slightly lower than 0.02%.

In summary, it appears that transfer learning could provide a more desired trade-off between the amount of training time and accuracy, especially when there is a need to train deep-learning models for a large number of programs in a scalable way.

7.6. Could incremental learning be leveraged to reduce the cost of retraining when a program is updated?

If a protected program is updated (e.g. patched), then there could be new instruction sequences available for attackers to utilize. Thus, we design an experiment to see whether DeepReturn could be trained using incremental training. In particular, we want to see if incremental training could significantly reduce the cost of model retraining. It should be noticed that this evaluation question is clearly related to the practicality of the DeepReturn approach.

In our experiment, the data of proftpd are divided into two parts. To simulate the software update scenario, one major part of the data will hold over 90% of all data. The other minor part is then considered as the new instruction sequences from patching, which is significantly smaller. We conducted the experiment with two different splitting ratios: 90%–10% and 95%–5%, respectively.

After we train the model from the scratch using majority of the data, the weights in the convolutional layers as well as the parameters in the batch normalization layers will be fixed. Therefore, when we do incremental training later, only weights in the dense layer of the neural network are updated to speed up the training process. This is a very common technique in incremental training.

As shown in Table 5 and Table 6, the result shows that the performance is neither downgraded nor improved much, which implies that incremental training for software updates is feasible. In our experiment, since only dense layer(s) need to be incrementally trained, the training time for newly added data is about 88% less than the time needed for training first time from scratch. Therefore, we conclude that for minor software updates, incremental training could be an ideal and cost-effective solution.

Using unbalanced data to train a neural network could cause the neural network to learn features that are not expected to be learned. For example, if a binary classification dataset contains 10% label 0 and 90% label 1, then the model can just learn to predict 1 and still achieve 90% accuracy. As a result, the rule of the thumb for training a neural network classifier is to ensure the dataset is balanced. However, a unique characteristic of the problem of detecting ROP payloads is that generating benign data samples is magnitudes harder and costlier than generating malicious data samples. And this unique characteristic indicates that there is actually a trade-off between how balanced the datasets are and how practical (e.g., how costly) to apply deep learning to detect ROP payloads. To gain a deeper understanding of the trade-off, we conduct an experiment to compare the performance of a model A trained with a balanced dataset and a model B trained with an unbalanced dataset (i.e., using the same amount of benign data samples used to train model A but more malicious data samples than those used to train model A).

We use proftpd data for this experiment. Suppose we only have 19,500 (benign) gadget-like instruction sequences available and it is too costly to generate more, then we use more than 19,500 (malicious) real gadget chains when training a model. We conduct the experiment 4 times with different levels of unbalance to investigate how the model’s performance is affected by levels of unbalance. The ratios of the number of gadget-like-instruction sequences, which is always 19,500, to the number of real gadget chains are set to be 0.4, 0.6, 0.8, and 1.0, respectively.

We use five essential metrics when training on unbalanced data: accuracy, false positive rate (FPR), false negative rate (FNR), F1-score, and area under receiver operating characteristic curve (ROC-AUC). A good classification model should do well in all these metrics. Note that we do not include figures of ROC curve here because the ROC-AUC is always over 0.9 for most of the cases.

The results are shown in the Table 7. The accuracy is increasing as more malicious data samples are used in training, but the FPR is in general getting worse. F1 score and ROC-AUC have the same trend as accuracy, which has increased performance when more malicious data samples are used.

The results have several implications. First, when the number of benign data samples cannot be further increased, neither using balanced data nor using unbalanced data is a clear “winner”. Each strategy has pros and cons. Second, if the security officer is most concerned with the FPR, then he or she will need to be careful in using unbalanced data. In particular, the security officer should determine whether the trade-off between the FPR and the other 4 metrics is worth it. Third, if the security officer is most concerned with any of the other 4 metrics, then using unbalanced data can be a good strategy.

We now consider the throughput of DeepReturn on nginx. We use throughput, rather than latency, as the performance criteria because DeepReturn is an IDS and it does not block the traffic. In other words, DeepReturn exerts zero runtime overhead for the protected program.

We can calculate from Table 2 that the disassembler works at a speed of 665 Mb/s, which can match the traffic on a server that is not too busy. Moreover, since the ASL-guided disassembly can be parallelized very well, we can split the workload across multiple servers running DeepReturn, further increasing the throughput.

The CNN in DeepReturn can classify 27k potential gadget chains in one second. We observe the number of potential gadget chains is rather small. On average, in the 665 Mb data processed within one second, only 13 potential gadget chains are fed into the neural network for classification. Thus, the performance of the entire IDS is bounded by the speed of disassembly and the overall throughput is 665 Mb/s.