Risk-aware multi-objective optimized virtual machine placement in the cloud

Abstract

Cloud computing, while becoming more and more popular as a dominant computing platform, introduces new security challenges. When virtual machines are deployed in a cloud environment, virtual machine placement strategies can significantly affect the overall security risks of the entire cloud. In recent years, the attacks are specifically designed to co-locate with target virtual machines in the cloud. The virtual machine placement without considering the security risks may put the users, or even the entire cloud, in danger. In this paper, we present a comprehensive approach to quantify the security risk of cloud environments from network, host and VM. Accordingly, we propose a Security-aware Multi-Objective Optimization based virtual machine Placement scheme (SMOOP) to seek a Pareto-optimal solution that reduces the overall security risks of a cloud, while considering workload balance, resource utilization on CPU, memory, disk, and network traffic. New placement strategies are designed and our evaluation results demonstrate their effectiveness. The security of clouds could be improved with affordable overheads. The latest VM allocation policies are further studied and integrated into our designs to defeat the co-residence attacks.

Keywords

Cloud security multiple objective virtual machine placement risk metrics model VM allocation strategy

1. Introduction

Cloud computing is the basis of many services in our daily life, such as email services and file sharing services. In an Infrastructure as a Service (IaaS) cloud like Amazon EC2 [4], many virtual machines (VMs) share a physical server. The placement of virtual machines can employ different strategies, leading to different computing performance, energy consumption, and resource utilization. Therefore, given different resource constraints, how to achieve multiple objectives is a very important problem in cloud computing. Such a problem has attracted extensive attention recently [9,27,30].

With resource and other constraints, the virtual machine placement (VMP) is essentially a multiple objectives optimization problem. Phan et al. [30] used an Evolutionary Multi-Objective Optimization (EMOA) algorithm to build Green Clouds when considering energy consumption, cooling energy consumption and user-to-service distance in the VMP optimization. Xu and Fortes [39] proposed a genetic algorithm with fuzzy multi-objective evaluation to minimize the total resource wastage, power consumption and thermal dissipation costs in VM placement. Shigeta et al. [36] suggested to assign different weights to multi-objective on cost and performance and built a cost evaluation plug-in module to search for the optimal VM placement. Some other research focused on minimizing the overall network cost while considering the communication requirements [3,29], or applying the constraint programming (CP) engine to optimize VMP [2,12]. While these multi-objective optimization placement schemes greatly improved the overall performance of the cloud, the security risk of the entire cloud environment was not considered as an objective or at most considered as one constraint in the initialization phase.

At the same time, there are new types of attacks [7,14,15,19,26,31,32] targeting the cloud infrastructure. In consequence, the security risk exposed to the user depends not only on how secure the VM itself is, such as the operating system and applications running inside, but also on the Virtual Machine Monitor (VMM or Hypervisor), running underlying the VMs, and other VMs co-residing on the same node.

We believe that security should be considered as one key objective, the same as the energy and performance, in VM placement. In our previous work [27], we proposed a VMP scheme based on the security risk of each VM. However, the security analysis of our previous work mainly focused on dependency relations. Yuchi and Shettey [43] extended our previous work to the VM placement initialization. Yu et al. [42] proposed isolation rules to formulate the VMP behavior based on the Chinese wall policy. Unfortunately, this work mainly focused on improving security and overlooked other objectives, such as energy saving and resource utilization. Besides, the security measurements in this work mainly considered the vulnerabilities of VMs, the hypervisor, or security regulations, without considering the security assessment of a VMP.

When comparing different VMP schemes, the security metrics can only be evaluated after a placement is specified. For example, a specific placement scheme has a unique attack path exposed by co-residence that may disappear in a different placement. Therefore, there is no generic function to map a placement scheme into a security assessment value. We cannot simply apply any existing evolutionary multi-objective optimization algorithm (EMOA) to solve our problem directly. Furthermore, the low efficiency and the complicated security assessment require us to design our own crossover and mutation procedures in the EMOA algorithm.

In the past few years, co-residence attacks have attracted a lot of attention. The straightforward solution to such attacks is to eliminate the side channel directly [5,35,38]. However, this solution requires the modification of the hardware or the cloud platforms, which is not practical. Most recent work [6,20,21] tried to use the VM allocation policy to make the co-residence infeasible to achieve. Motivated by their designs, we also aim to integrate these objectives into our design in order to mitigate co-resident attacks.

To this end, in this paper, we propose a VM placement specific security measurement of the cloud, and a new VMP approach to provide better intrusion resilience, workload balance, resource utilization, and network performance. In the proposed VM placement specific security assessment, we consider the vulnerabilities not only on VMs and the hypervisor themselves, but also the co-resident VMs and the network connections that will be changed with the VM placement. Based on the proposed security measurement scheme, we design an evolutionary multi-objective optimization algorithm, named as Security-aware Multi-Objective Optimization based virtual machine Placement algorithm (SMOOP), to seek a Pareto-optimal solution balancing the multiple objectives on security, resource utilization, network traffic and workload.

Our proposed scheme features an innovative combination of the following contributions.

We conduct security assessment of the cloud from four aspects: networking, co-residence, hypervisor vulnerabilities, and VM vulnerabilities. The proposed security risk assessment is placement specific and crosses multiple dimensions. We provide detailed metrics and an approach to measure the security of the cloud in the case study and experiments.

We consider security as one objective in VMP strategies, with other objectives and constraints at the same time. To the best of our knowledge, this is the first work that includes a placement specific security assessment in the context of multi-objective optimization based VMP.

We propose a highly scalable scheme, SMOOP with five placement strategies, to seek the Pareto-optimal placement to balance multiple objectives. Users could adjust the weight list according to their own preferences. The experimental results confirm the effectiveness of our strategies and SMOOP can provide an overall improved security of the cloud with reasonable overhead.

We study the latest VM allocation policies and integrate them into our design to defeat co-residence attacks. Thus, our algorithm is compatible with up-to-date VM allocation policies. Besides, our placement strategies can be configured by the users to fit for different business needs.

The rest of the paper is organized as follows. In Section 2, we discuss the related work. Section 3 describes the formulation of the VMP optimization problem. Section 4 describes the design and implementation of SMOOP. The evaluation results are discussed in Section 5 and Section 6 summarizes our work.

2. Related work

As cloud computing becomes more popular, virtual machine placement (VMP) has become one of the most critical problems in clouds. Several studies have observed the similarity between the VMP problem and the bin-packing problem [28]. There are some simple but effective heuristics for this problem, such as First Fit, Best Fit, and Worst Fit. Based on these policies, First Fit Decreasing (FFD) could improve the approximation ratio if the objects are first sorted in decreasing order of their weights. If OPT denotes the optimal number of bins, then FFD is guaranteed to use no more than 11/9 OPT + 6/9 bins [17,18].

As time passed, new types of attacks targeting the cloud infrastructure appeared. For example, some attacks, such as those discussed in [14,15,19,32], exploit vulnerabilities of the hypervisor (or Virtual Machine Monitor, VMM), e.g., Xen [7] or KVM [26]. Once an attacker compromises the hypervisor, he or she can take over all the VMs running on it. In [31] (the HYG attack), the initial stage of the attack is to locate a target VM. Upon success, the attacker will try to launch a VM on the same physical server. It is a placement based attack and the success of the attack depends on the placement strategies of the cloud, or the configuration policy of the cloud. Apparently, collocating with vulnerable virtual machines, or “bad neighbors”, on the same physical server does increase the security risks of the cloud users. Thus, a lot of research on cloud computing has set the goal to improve the security level of data centers [1]. At the same time, some existing research on the co-residence based attacks, e.g., side channel attacks, demonstrates the real threat to the normal users if they are co-located with a vulnerable or malicious VM [31,40,41,45]. Thus, the security aware VMP has been investigated as a practical solution to mitigate such attacks [2,27,34].

Saeed et al. [2] presented a security-aware approach for resource allocation in clouds which allowed for effective enforcement of defense-in-depth for cloud VMs. Ravi et al. [24] proposed a framework that allowed providers to impose restrictions on the allocations to be made to their hosts and users to express constraints on the placement of their virtual machines (VMs). Both of them tried to enhance the security level by modeling the cloud provider’s constraints or customer’s requirements as a constraint satisfaction problem (CSP). However, the placement generated by these methods can only satisfy the input constraints, rather than being an optimal placement to meet multiple objectives.

Some other research utilized isolation rules in the VMP. Afoulki et al. [1] proposed a VMP algorithm which improved the security of clouds by performing isolation among users. Each user can submit a list of adversary users with whom it does not want to share a physical machine. Yu et al. [42] also proposed isolation rules to formulate the VMs placement behavior based on Chinese wall policies.

Our previous work [27] proposed a VM placement scheme based on security risk of each VM, and Yuchi and Shettey [43] extended it to the VM placement initialization. Both of them mainly focused on the dependency relations. Yuchi and Shettey’s method also oversimplified the problem and did not reflect the potential risk caused by co-resident VMs, whose importance was discussed in [37,44]. In [44], the author studied the characteristics of different PaaS cloud and the co-resident threat in placement policies. They implemented a memory-bus based covert-channel detection for co-residence and presented an efficient launch strategy. Their experiment concluded that the risk caused by co-residency was real in popular PaaS clouds. Previously, we have investigated to periodically migrate VMs based on the game theory, making it much harder for the adversaries to locate the target VMs in terms of survivability measurement [46]. But we did not consider the risk caused by the co-resident VMs in the same physical machine.

Our work in this paper differs from the existing research mainly in two aspects. First, existing work simplifies the security consideration in the placement. They mainly consider the security constraints or regulations, or vulnerabilities of the VMs or hypervisor in the placement. They often overlook co-residence attacks, which is a key factor in VM placement. In our security-aware VMP, we comprehensively consider security assessment associated with placement, including the security risks in the network connection, co-residence, VMs and hypervisor. Second, existing work often emphasizes on security while overlooking other performance factors. We propose an optimal solution satisfying multiple objectives on security, resource utilization, and network traffic.

To the best of our knowledge, there are only two existing solutions using VM allocation policies to defeat the co-resident attacks. In [6], the authors proposed a Co-Location Resistant (CLR) algorithm. All servers are labelled either in open or closed state and can switch the state as needed. Open (closed) means the server can (cannot) receive more VMs. CLR would try to maintain a fixed amount ( $N_open$ ) of servers staying open state, and allocate a new deployed VM to one of these servers randomly. If the allocation procedure failed, the server will be switched into the closed state, and another inactive server will be activated and marked as open to receive new deployed VM. Specifically, the larger $N_{open}$ is, the more co-location resistant the algorithm becomes. Therefore, the best case scenario is that all servers are open. Han et al. [20,21] proposed a Previous-Selected-Server-first (PSSF) allocation policy to increase the difficulties of co-locating with target VMs and decrease the attack efficiency. The workload balance and power consumption are considered in the allocation as well. They defined and modeled some security metrics to assess the attack and compared the difficulty of achieving co-residence between their PSSF and CLR. They implemented the experiments on the simulation environment CloudSim [11,13]. Per our study, both allocation policies can be adapted into our mutation operation with minor modification.

3. Problem formulation

In this section, we describe our system and metrics to model the objectives, and constraints of virtual machine placement in a cloud.

3.1. Threat model and security assumptions

In this paper, we mainly consider co-residence based attacks, such as cross-VM side channel attacks. Also, we assume that the attackers are capable of utilizing vulnerabilities in both VMs and virtual machine monitors (VMMs, or hypervisor) of the clouds.

We have the following assumptions for the cloud: 1 ◯ the cloud management, placement related software components, and the migration process are all secure; 2 ◯ for simplicity, each migration of a VM will result in affordable cost in terms of service interruption and consume the same amount of resources; 3 ◯ the cloud provider has enough CPU, network bandwidth, and other resources to perform arbitrary migration of VMs; and 4 ◯ the cloud provider has sufficient resources as the reward, e.g., extra memory or CPUs, to incentivize VM migrations. The above assumptions ensure that a change of VM placement is both acceptable and affordable for cloud provider and clients.

3.2. Security assessment

In a cloud, an attacker can compromise a VM through different attack paths. They can compromise a VM through the vulnerabilities (in the operating system, or applications) carried by the VM, the co-resident VMs, the host VMM, or VMs on different physical machines having network connections with. Therefore, we cannot simply use the vulnerabilities of VMs, or the vulnerabilities of the hypervisor to evaluate the security risk of an entire cloud. We need a comprehensive approach to measure the security risks of a specific placement scheme.

Fig. 1.

Security risk metrics.

For this purpose, we propose a four dimensional security risk evaluation model, as shown in Fig. 1, to assess the security risk of a cloud. The new evaluation model covers all possible attack paths in a cloud. Four different types of security risks are described as follows.

VM risk ( $R_{1}$ in the figure): the risk/vulnerability carried by a VM itself. If a VM has more vulnerabilities than others, it is more likely to be compromised first. Vulnerable VMs can be used as stepping stones to attack co-resident VMs and the underlying hypervisor to gain more privileges.

VMM/hypervisor risk ( $R_{2}$ ): the risk/vulnerability carried by a VMM/Hypervisor. An adversary may gain the administrative privileges via the vulnerabilities in a hypervisor or the control VM. Such vulnerabilities will enable the adversary to compromise all guest VMs on the hypervisor.

Co-residence risk ( $R_{3}$ ): the risk caused by the VMs co-resident on the same hypervisor. Assume that, in the figure, VM1 (an attacker VM) and VM2 (a normal user VM) share the same CPU core or are located on the same physical machine, the attacker will be able to steal the user’s private information, such as the cryptographic key, via side-channel attacks.

Network risk ( $R_{4}$ ): the security risk of a VM caused by the network connections. For example, VM1, located in host 1, provides web services, and the VM2, located in host 2, contains a database server. The attacker may compromise the database server through accessing the web server, e.g., SQL injection.

3.3. Explanation of the security metrics

Using the proposed security risk assessment model, we can assign or calculate the values of each type of the risks based on specific hardware, software, and network configuration. In this section, we provide an example to show how to quantify the values of each type of security risks, and also how to calculate the overall security risk of the entire cloud. In the example, we assume we have N VMs and M physical machines.

$R_{1}$ : CVSS (Common Vulnerability Scoring System) is a popular tool to measure the vulnerabilities of software or hardware [27]. We can use vulnerability scanner tools, such as Nessus and Qualys, to generate the vulnerability list for every VM. Based on the list, we can score each VM’s risk carried by itself. One way to do that is to simply use the worst vulnerability’s score, with the assumption that the vulnerable level of a VM is not higher than the worst vulnerability of that VM, although there certainly exist more refined approaches. Also, it could be calculated by other security metrics proposed for physical hosts [12].

The CVSS score uses an interval scale of $(0, 10)$ to measure the severity of vulnerabilities. If different tools are used to score vulnerabilities, we can choose CVSS as our baseline to normalize all vulnerability scores. For example, if a different tool scores 7 for a vulnerability which is scored 8 in CVSS, then the correlated score relationship is $7 : 8$ between the tool and CVSS. At the same time, for some vulnerability score tools, such as Nexpose, CVSS is considered as a major factor in the score calculation model.

For a VM $v_{i}$ , its VM risk $R_{1}$ is ${VM}_{R_{1}}^{i} = {SCORE}_{i} / 10$ , so its range would be limited to a scale of $(0, 1)$ . Note that $R_{1}$ is not affected by a specific placement.

$R_{2}$ : The risk level of a hypervisor is determined by two factors: its own vulnerability and the VMs running on it. For hypervisor’s own vulnerability, we can use scanner tools to generate the vulnerability list and also use the most severe one to obtain the ${SCORE}_{hypervisor}$ from CVSS. We use ${Risk}_{hypervisor} = {SCORE}_{hypervisor} / 10$ to indicate the vulnerability of the hypervisor so that the value is in 0 to 1, similar to $R_{1}$ .

There are different ways to calculate how the guest VMs can affect the security of hypervisors. In this paper, we mainly consider the VM with the highest risk since this may be the most vulnerable attacking surface to the hypervisor. Assume VM $v_{i}$ is on the host K, its hypervisor risk $R_{2}$ is calculated as: $R_{2}^{i}$ = ${Risk}_{hypervisor} (1 + max (R_{1}^{j} p_{j K}))$ , where $j = 1$ to N, and $p_{j K} = 1$ if VM $v_{j}$ is placed on host K as well. Different from $R_{1}$ , $R_{2}$ is affected by different VM placements.

$R_{3}$ : A malicious VM may compromise a normal VM if they are co-located on the same physical machine. If an attacker compromised a VM, he/she can compromise, with enough time, other co-resident VMs eventually. So a VM can survive only if all other co-resident VMs can survive. Obviously, the risk rate of a co-resident VM will be affected by its network connected parties. This will be a dead circle until either one is determined and we choose to set the co-residence risk first. For a VM $v_{i}$ on the host K, its co-residence risk $R_{3}$ is calculated as: $R_{3}^{i}$ = $1 - \prod_{j = 1}^{N} (1 - R_{1}^{j} p_{j K})$ , where $p_{j K} = 1$ if VM $v_{j}$ is placed on the physical machine K. Similar to $R_{2}$ , $R_{3}$ will change if the VM placement changes.

$R_{4}$ : If an attacker compromised a VM, he/she is able to compromise (with enough time) all other VMs with network connections to the compromised VM. Considering the multiple layer connection in the network, given a VM, the depth first algorithm can be applied to generate all possible attack paths. However, when a loop exists, the start point of risk calculation can be hard to be determined. We aim to identify the correlated risk level between two VMs. So we design our algorithm aiming to improve efficiency, while maintaining the correct correlations. We only consider the risk caused by direct network connections in the paper. Thus, for a VM $v_{i}$ , its network risk $R_{4}$ is $R_{4}^{i} = 1 - \prod_{j = 1}^{N} (1 - R_{1}^{j})$ , where $v_{j}$ is a VM sending packets to $v_{i}$ directly, and $v_{i}$ and $v_{j}$ are not on the same host. $R_{4}$ changes with different VM placements as well.

With all types of risks defined as above, we define the security risk, $R^{i}$ of a VM $v_{i}$ as the following. $\begin{matrix} (1) & R^{i} = 1 - (1 - R_{1}^{i}) (1 - R_{2}^{i}) (1 - R_{3}^{i}) (1 - R_{4}^{i}) \end{matrix}$ Considering all simplicities applied above, our metrics would represent the correct risk level correlation between two VMs. For example, a VM with a risk rate of 70% is safer than a VM with a risk rate of 80%, however without knowing how much safer to be exact.

Assurance level of a physical machine. Once we have the security risk of each VM, we can further calculate the assurance level of a physical machine J using the following equation. $\begin{matrix} (2) & A L_{J} = 1 - \overline{(R^{i})} \end{matrix}$ where $V M_{i}$ ( $R^{i}$ is corresponding R value for $V M_{i}$ ) is located in physical machine J.

In general, the higher the assurance level is, the more secure physical machine will be, and the harder it is to be compromised by the attacker. However, the network communication from physical machines with lower assurance level to physical machines with higher assurance level may open an attacking channel for the attacker. Therefore, we should minimize the traffic between the low assurance physical machines and the high assurance physical machines. Besides, it can also be used as related information for potential cascade vulnerability correction [10,22,33]. Note that our algorithm does not include the clearance level of information stored in each physical machine.

3.4. Objectives in VM placement

Assume that we have N VMs and M physical machines. There are four values to optimize: security risk (SR), resource wastage (RW), network traffic (NT) and workload balance (WB). Our goal is to find solutions to minimize these values.

Security risk. Minimizing the security risk of the entire cloud is our first objective. The security risk of a VM $v_{i}$ is $R^{i} = 1 - (1 - R_{1}^{i}) (1 - R_{2}^{i}) (1 - R_{3}^{i}) (1 - R_{4}^{i})$ . To evaluate the security risk of the entire cloud, we need to consider the security risk of all VMs in the cloud. In security assessment, we use the median value of all VMs’ risk values as the risk level of the cloud. Thus, the security risk is calculated as follows. $\begin{matrix} (3) & f_{S R} = \overline{(R^{i})} \end{matrix}$ where $i = 1$ to N. The reason to choose the median value is twofold. First, the median value is more robust than the mean value. Second, in our placement generation, a dangerous VM will be isolated from other VMs. Thus, VMs with high risk values are outliers in our system. It is the same for VMs with low risk values.

Resource wastage. Minimizing resource wastage, while complying with the constraints, is the second objective in the VMP optimization. In this paper, we consider the wastage of multiple resources, including CPU, memory, and disk. Instead of using one value to measure the resource wastage, we use a vector to represent the resource wastage.

Assume the CPU, memory and disk capacity for a host J as $⟨ {CPU}^{J}, {MEM}^{J}, {DISK}^{J} ⟩$ . A VM $v_{i}$ requests resources as $⟨ {cpu}_{i}, {mem}_{i}, {disk}_{i} ⟩$ , therefore, the CPU wastage of the host J is $W_{J}^{c} = ({CPU}^{J} - \sum_{i = 1}^{N} {cpu}_{i} p_{i J}) / {CPU}^{j}$ , where $p_{i J} = 1$ if VM $v_{i}$ is placed on host J, otherwise it is 0. The memory wastage of host J is $W_{J}^{m} = ({MEM}^{J} - \sum_{i = 1}^{N} {mem}_{i} p_{i J}) / {MEM}^{J}$ . The disk wastage of host J is $W_{J}^{d} = ({DISK}^{J} - \sum_{i = 1}^{N} {disk}_{i} p_{i J}) / {DISK}^{J}$ .

For a physical machine J, we choose the maximum value from ${W_{J}^{c}, W_{J}^{m}, W_{J}^{d}}$ to represent the resource wastage of host J. We would like to minimize the total amount of the resource wastage of the entire cloud. $\begin{matrix} (4) & f_{R W} = \sum_{J = 1}^{M} max (W_{J}^{c}, W_{J}^{m}, W_{J}^{d}) \end{matrix}$ while subject to the following capacity constraints in each host J: $\begin{array}{l} (5) & \sum_{i = 1}^{M} {cpu}_{i} \times p_{i J} < {CPU}^{J} \\ (6) & \sum_{i = 1}^{M} {mem}_{i} \times p_{i J} < {MEM}^{J} \\ (7) & \sum_{i = 1}^{M} {disk}_{i} \times p_{i J} < {DISK}^{J} \end{array}$

Network traffic. The third optimization objective is to minimize the network traffic in cloud. One way to reduce the network traffic is to identify correlated VMs that exchange high volume of data with each other, and then put them on the same physical machine if possible. We use the following equation to measure the network traffic from VM $v_{i}$ to VM $v_{j}$ : $\begin{matrix} (8) & T_{i j} = P_{i j} / t \end{matrix}$ where $P_{i j}$ is the number of packets sent from VM $v_{i}$ to VM $v_{j}$ in time period of t. Therefore, the total network traffic in the time period of t is: $\begin{matrix} (9) & f_{N T} = \sum_{j = 1}^{N} \sum_{i = 1}^{N} T_{i j} g_{i j} \end{matrix}$ where $g_{i j} = 0$ if VM $v_{i}$ and VM $v_{j}$ are placed on the same host, otherwise it is 1.

As we discussed earlier, we should minimize the network communication from the low assurance physical machines to the high assurance ones. We use the following equation to measure the related traffic from physical machine I to J: $\begin{matrix} (10) & T M_{I J} = P M_{I J} / t \end{matrix}$ where $P M_{I J}$ is the number of packets sent from physical machine I to J in time period of t. Therefore, the network traffic from the low assurance physical machines to the higher ones in the time period of t is: $\begin{matrix} (11) & {NonSecure}_{N T} = \sum_{j = 1}^{N} \sum_{i = 1}^{N} T M_{I J} g_{I J} \end{matrix}$ where $g_{I J} = 1$ if physical machine I has a lower assurance level than that of J, otherwise it is 0.

Workload balance. The fourth optimization objective is to maintain the workload balance in the cloud. The importance of workload balance is twofold [23]. For cloud providers, evenly distributing VMs helps to decrease the probability of over-utilizing specific servers. The best scenario is that all physical machines are active and workloads are distributed evenly among them. However, that will be conflicting with the goal of energy consumption saving. Thus, in our algorithm, we focus on spreading workload evenly among active physical machines after the number of active physical machines is determined. For the customers, they may prefer that their VMs are not all located together on the server for better survivability and availability. So far, our algorithm does not consider this requirement, but it could be easily integrated into our algorithm according to the users’ preferences.

In cloud computing, over-subscription is a common practice which allows the service providers to allocate more resources to users than the servers’ actual capacity [8]. As the most critical resource, the over-subscription of CPU capacity is always enabled, which would make the RAM capacity a bottleneck. In this paper, we choose the RAM usage level as the criteria of workload balance measurement.

Assume the memory usage of the host j is ${Mem}^{j} = \sum_{i = 1}^{N} {Mem}_{i} \times p_{i j}$ , where $p_{i j} = 1$ if VM $v_{i}$ is placed on host j, otherwise it is 0. $P M_{a}$ represents the number of active physical machines in a placement. The average memory usage in all active physical machines will be: $\begin{matrix} (12) & {Mem}_{m} = \sum_{j = 1}^{N} {Mem}^{j} / P M_{a} \end{matrix}$

The variance of the workload balance of all active physical machines can be calculated using the following equation. $\begin{matrix} (13) & f_{W B} = \sqrt{(\sum_{j = 1}^{N} {({Mem}^{j} - {Mem}_{m})}^{2}) / P M_{a}} \end{matrix}$

In our placement strategy, we consider the security risk, resource wastage, network traffic and workload balance as the placement objectives. Note that our system does not limit the number of objectives or constraints. The users can add more objectives or constraints, such as energy or migration cost, based on their preferences.

4. SMOOP design

With the proposed security metric of VMs, we can quantify the risk level of a cloud. As a typical multi-objective optimization problem, the objectives may conflict with each other. For example, if we place more VMs on a physical server, it will be less secure due to the co-residence problem. However, it can reduce the resource wastage and network traffic. It is impractical to always find the optimal solution minimizing all objectives. The evolutionary multi-objective algorithms (EMOA), such as NSGAII [16], are popular solutions to such multi-objective optimization problems. Using EMOA, we can obtain Pareto-optimal solutions balancing the objectives of security, network traffic, resource utilization and workload balance.

Challenges. In the bin-packing problem, objects of different volumes must be packed into a finite number of bins of volume V in a way that minimizes the number of bins used. The Virtual Machine Placement (VMP) can be considered as a bin-packing problem, where each VM needs to be placed on a physical server once and only once, with multiple dimensions of volume constraint, such as CPU, Memory and Disk. As a regular bin-packing problem, VMP is also a $N P$ hard problem [25]. The challenge is that the security metrics can only be evaluated after the placement is specified. For example, in a specific placement scheme, a unique attack path exposed by co-residence may disappear in a different placement. Therefore, there is no generic function mapping a placement scheme into a security assessment value. As a result, we cannot use any existing multi-objective programming solutions to solve our problem. Furthermore, we have complicated security strategies in each placement generation, and we have to design a new crossover and a new mutation procedure in the EMOA algorithm.

4.1. Security-aware multi-objective optimization based VMP

In this section, we present our Security-aware Multi-Objective Optimization based virtual machine Placement (SMOOP). The algorithm is shown in Algorithm 1. Table 1 describes the variables used in the algorithm.

Algorithm 1

SMOOP

In practice, FFD (First-Fit with the possible fullest node) has been widely used in VMP. It can quickly provide a placement with consideration on resource utilization. Thus, we use it to generate a baseline for future comparison in the algorithm. As shown in Algorithm 1, SMOOP generates hundreds of placements and passes those with high fitness value to the next iteration. In each iteration, randomly chosen parents are applied to crossover and mutation operations. An elite choosing function is designed to improve efficiency. For each generated temporary placement in an iteration, we apply a multi-objective evaluation function to assign ranking values. The highly ranked placements are put into a candidate pool, and used as the parents for next iteration. The preference of multi-objective evaluation can be adjusted (described in Section 4.3) in our algorithm.

Table 1

Variable definition

Variable	Description
V_N	Number of Virtual Machines
P_N	Number of Physical Machines
N_G	Number of iteration
N_IS	Number of placement in candidate pool
N_E	Number of Elite would be passed to next iteration
N_C	Times of Crossover operation in one iteration
N_M	Times of Mutation operation in one iteration
OFF_C	Offspring generated by Crossover Operation
OFF_M	Offspring generated by Mutation Operation

In the initialization phase, the consideration for the migration cost can be avoided. Our goal is to search for the best possible placement plan based on the multi-objectives requirement. The crossover operation is used to improve the overall efficiency. In the re-optimization phase, which is triggered by adding VMs or removing VMs, the migration cost is considered as an important factor in the mutation operation to limit the number of migrating VMs (in switch() function of Algorithm 3). Note that our goal is to improve the survivability of entire cloud, so we do not optimize our solution for a specific VM. Therefore, our approach may lower the security level of a specific VM, while the overall security level of the entire cloud can still be improved.

4.2. Crossover and mutation operation

The crossover operation, shown in Algorithm 2, is one of the key elements in our algorithm. The main purpose of the crossover operation is to guarantee that there is always a chance to generate newly improved placement based on the existing placement in the current iteration.

Algorithm 2

Crossover(X, Y)

4.2.1. Isolated zones

Since the security is a key factor in the placement generation, we introduce isolated zones in our algorithm to accommodate different security demand. Physical machines with the highest hypervisor risk levels are put into isolated zones. The most dangerous VMs and VMs connected to them are placed into the isolated zones by priorities. The purpose of the isolated zones is to isolate the most dangerous VMs first and reduce the number of attack paths through network connections.

If all physical machines use the same copy of hypervisor, the vulnerabilities of all the hypervisors will also be the same. In such a situation, we have the following assumptions. 1 ◯ The possibility to compromise any physical machine through the hypervisor attack surface for a specific VM is the same. 2 ◯ If the communication bandwidth between two VMs is larger than zero, the possibility to compromise one VM through another VM will be non-zero.

We propose five security related strategies to reduce security risk during each placement generation.

Placement strategy I: Deploying a VM into a physical machine which has network connections with it.

Placement strategy II: The high risk VMs should be deployed into the isolated zones.

Placement strategy III: The low risk VM without any connection with VMs in isolated zones should be deployed into low risk physical machines. Strategy II and III generate physical machines that contain only low risk VMs and have no network connections with high risk VMs in isolated zones.

Placement strategy IV: The marked lowest and highest hypervisor risk physical machines should have a higher probability to be kept during crossover operation. This is based on our strategy II and III.

Placement strategy V: If a VM on one physical machine has connection with a VM on a different physical machine, we should migrate one of them to the same physical machine.

We use Placement strategy I as an example to explain how the placement strategy can reduce the security risk. The motivation of this strategy is to reduce $R_{4}$ caused by network connections.

Assume that physical machine $P M_{I}$ already has a set of VMs, $S_{i} = (v_{a}, \dots, v_{i})$ and $P M_{J}$ has a set of VMs, $S_{j} = (v_{b}, \dots, v_{j})$ . There is no network connections between $S_{i}$ and $S_{j}$ . When a new VM $v_{n}$ is to be placed and it has network connections with at least one VM in $P M_{I}$ . If $v_{n}$ is placed into $P M_{I}$ , $S_{j}$ on $P M_{J}$ will not be affected by attacks through network connections. It will only affect $S_{i}$ on physical machine $P M_{I}$ . For any VM $v_{i} \in S_{i}$ , $R_{3}^{i}$ will be updated by adding the new VM $v_{n}$ . Assume that the current co-residency risk of $v_{i}$ is $Old (R_{3}^{i})$ , then we have $\begin{matrix} (14) & New (R_{3}^{i}) = Old (R_{3}^{i}) + (1 - Old (R_{3}^{i})) R_{1}^{n} \end{matrix}$ where $R_{1}^{n}$ is the risk level of $v_{n}$ .

If $v_{n}$ is placed into a physical machine $P M_{J}$ , not only will $R_{3}$ be updated, but $R_{4}$ introduced by network connections will also be increased by $\sum_{v = c}^{K} T_{n v}$ (all traffic through VM $v_{n}$ to connected VMs on physical machine $P M_{I}$ ).

According to the security metrics defined earlier, in this case, the co-residence risk of each VM $v_{j} \in S_{j}$ will be $\begin{matrix} (15) & New (R_{3}^{j}) = Old (R_{3}^{j}) + (1 - Old (R_{3}^{j})) R_{1}^{n} \end{matrix}$ Also, $R_{4}$ of VM $v_{j}$ increases as well. $\begin{matrix} (16) & New (R_{4}^{j}) = Old (R_{4}^{j}) + (1 - Old (R_{4}^{j})) R_{1}^{n} \end{matrix}$

Therefore, following strategy 1, allocating VM $v_{n}$ on to $P M_{I}$ rather than $P M_{J}$ can reduce security risk. Other placement strategies can reduce the security risk as well.

In our implementation, Strategies I, II, III are applied for VM deployment and Strategies IV and V are applied for crossover and mutation procedure. When a VM need to be deployed, $⟨ VM, PM ⟩$ pairs would be generated, in which PM would be all available ones. Each pair has an associated “strategy fit set”; if a given VM placement fits a particular strategy, then the strategy number will appear in this set. Each strategy fit set will then be evaluated in order to decide a final deployment choice. To do so, the set will be tested to determine whether it satisfies each strategy, in order of weight, from highest priority to lowest priority. This process will progressively eliminate placement choices that do not satisfy a given strategy, successively shrinking the set of option. For example, given the strategy fit sets 1, 2, 1, and 2, 3, we examine these sets for membership of strategy 1. This eliminates set 2, 3, as it does not satisfy the highest priority strategy (strategy 1). We continue by testing the remaining sets for membership of strategy 2, and so on. This discards set 1, 2, as it does not satisfy the next-highest priority strategy (strategy 2). As a result, only set 1, 2 remains, indicating that its associated VM/PM deployment will be utilized. The selection rules (i.e., the system of weighting strategies) can be adjusted in real-time. For instance, after running for a period of time, it might be optimal for strategy 2 to be promoted to the highest priority; as a result, the weighting factors for strategies 1 and 2 would be swapped. Also, new strategies should be added, in response to the runtime environment. In this way, the system may evolve to fit real-world situations.

4.2.2. Mutation operation

Mutation operations, shown in Algorithm 3, operate on a randomly-chosen temporary placement, trying to obtain an improved result. Its purpose is to keep evolving the existing placement with limited migration cost.

Algorithm 3

Mutation(X)

When a Pareto-optimal solution is generated, our algorithm checks the workload balance in every physical machine, migrating marked VMs among physical machines. In the switch() function of the algorithm, a VM can only be switched (migrated) to another physical machine to which it has network connections. The strategy is to guarantee that random evolving will not jeopardize the isolation of dangerous VMs and reduce unnecessary switching operation.

4.3. Prioritize the objectives

In our current fitness function, we have four objectives, including minimizing the security risk, minimizing the resource wastage, minimizing network traffic and minimizing the variance of the workload balance. Our algorithm tries to provide a Pareto-optimal solution which can be as good as possible in every degree based on the four objectives. To enable users to prioritize the objectives according to their business preference, we can add weight factors into the fitness function. $\begin{matrix} (17) & f = w_{1} f_{S R} + w_{2} f_{R W} + w_{3} f_{N T} + w_{4} f_{W B} \end{matrix}$ where $w_{i}$ represent different weights and $\sum_{i = 1}^{4} w_{i} = 1$ . If we consider that the security is more important than the other objectives, we can assign larger weight on the security risk.

Currently, our algorithm can optimize and balance security, the utilization of CPU, memory and disk, the network traffic and workload. Our algorithm can be easily extended to support more objectives and constraints, such as energy consumption saving.

5. Evaluation

We implemented our solution in Java and conduct all experiment in a simulation environment. All input data are provided through configuration files. Multiple threads are used to improve the performance. We randomly generate a large number of VMs with different parameters to evaluate SMOOP. Every VM needs to be deployed into one physical machine, without considering the migration cost. In our evaluation, for each VM, we randomly assign the requirement of CPU, memory, and disk. The vulnerability score is assigned based on the uniform distribution. Following the same method, we configure the physical machine. The numbers of VMs and physical machines vary between experiments.

5.1. Computing complexity

Assume that there are M physical servers and a total of N VMs. Our algorithm iterates for k times with candidate pool size of P in each iteration. In our case, N is far larger than other factors here. The value of our objectives for each Virtual Machine Placement (VMP) could be calculated in bounded by $O (M N^{2})$ . Assume that the fitness value of a VMP can be calculated in constant time with a fitness function, which is $O (P)$ for the entire candidate pool. We sort the candidate pool with complexity of $O (P log P)$ . The elite choosing operation will take constant time. The crossover and mutation operation in our algorithm is bounded by $O (M N^{2})$ . The overall combined complexity of our algorithm is $O (k (M N^{2} + P M N^{2} + P log P))$ , thus $O (k P M N^{2})$ . If we consider the cascade effect of network, the computing time of risk value for each VM would be changed from $O (N^{2})$ to $O (N^{3})$ , which is caused by the construction of all possible attack paths with a Depth-First search. Then the complexity of our algorithm will be bounded by $O (k P N^{3})$ . In order to improve efficiency, we simplified the computing of risk value, while maintaining correct correlations. Also, multi-threading is used in our implementation, and we used 8 threads to conduct elite selection, crossover and mutation operation simultaneously.

Fig. 2.

Scalability.

We test our implementation in a 8 core processor with 16 GB memory. The overall performance of our algorithm is affected by the number of VMs, the number of physical machines, and the number of candidate placement generated in each generation. Figure 2 shows the computing time for each generation under the following setting: 1 ◯ 100 different placements are generated for each generation. 2 ◯ 270 operations are done in each generation. With 10000 VMs and 500 physical machines, each generation takes about 15–20 minutes. If we reduce the number of VMs to 3000, each generation takes about 2 minutes.

5.2. Effectiveness in risk reduction

The security risk is a key consideration in VMP. To evaluate if our strategies can improve the security level of the entire cloud, we conduct the experiments considering the risk level as the only objective in the placement. Figure 3 shows the security risk with 800 VMs and 60 physical machines.

Fig. 3.

Comparing with random-FFD.

At the beginning of each simulation, we always generate 100 placement with the random-FFD algorithm and use the lowest risk level as the baseline reference. We collect the placement with the lowest risk level in each generation. Within 20 generations, the risk level of the entire cloud can be reduced by 25% to 30%.

Fig. 4.

Non-secure network traffic.

As we discussed before, the network from low assurance level physical machines to higher ones would always be minimized to make the network more secure. Figure 4 shows the non-secure network traffic bandwidth is also dropped about 3% to 5% with other multiple objectives optimized.

Fig. 5.

Security improvement with different number of VMs.

Figure 5 shows the security risk with different number of VMs and physical machines in each generation. Despite the increased number of the VMs, the median value of the risk level of VMs is stable within the range of 0.82 to 0.84. If we check the placement with the lowest risk level in the first generation, our algorithm improves with the increased number of the VMs. We repeat our experiment 20 times with different numbers of VMs and physical machines. The reduced risk level is from 5% (400 VMs and 20 physical machines) to 15% (6400 VMs and 400 physical machines) just in the first generation.

5.3. Effectiveness in workload balance

In our algorithm, we focus on spreading workload evenly among active physical machines once the number of active physical machines is determined.

Figure 6 shows experimental results with weight setting: (0.4 (risk level), 0.4 (Workload Balance), 0.1 (Resource Wastage), 0.1 (Network Traffic)) in an environment of 400 VMs and 60 physical machines. Per the setting of our experiment, the memory request of a VM ranges from 4 GB to 40 GB and each physical machine could provide 256 GB memory. Figure 7 demonstrates the usage of memory in active physical machines with the best fitness value. VMs are aggregated into 31 active physical machines and the usage of memory in those physical machines achieves 92% and higher, which ranges from 237 GB to 255 GB.

Fig. 6.

Multi-objective optimization with workload balance.

Fig. 7.

Memory usage in cloud after balanced.

5.4. Effectiveness of multi-objective optimization

In this experiment, we consider multi-objectives on risk level, resource wastage, and the network traffic.

Fig. 8.

Multi-objective optimization.

Fig. 9.

Multi-objective optimization 2.

Figure 8 shows experimental results with weight setting (0.8 (risk level), 0.1 (Resource Wastage), 0.1 (Network Traffic)) in an environment of 800 VMs and 60 physical machines. The risk level has weight of 80%, resource wastage and network traffic have weight of 10% for each in the fitness function. We collect the placement with the best fitness value. The baseline is still the best placement chosen from 100 random-FFD placements. If a physical machine can hold hundreds of VMs, the placement generated by FFD will be using the minimum number of physical machines. With setting of (0.8, 0.1, 0.1), the active number of physical machines and resource wastage are limited, with much improved security.

We also run the experiment with weight setting (0.4 (risk level), 0.3 (Resource Wastage), 0.3 (Network Traffic)) in an environment of 3000 VMs and 200 physical machines, and the results are shown in Fig. 9. Since the resource wastage and network traffic have higher weights, the allowance of resource wastage was controlled and it also affects security improvement we can achieve. A cloud provider can always change the optimization preferences by changing the weights of different objectives.

5.5. Comparison with random-FFD algorithm

In the experiment, we use with 1600 VMs and 120 physical machines, and we generate 100 placements with the random-FFD algorithm. We choose the placement with the lowest median value of risk level. After running our algorithm to reduce the risk level, we choose the best placement. As shown in Fig. 10 and Fig. 11, we can see that the risk level of the entire VM set has been effectively reduced. In the figure, the X-axis is the risk level value of VMs. For example, $< 0.2$ means that the risk level is between 10% and 20%. With 1600 VMs, the risk level under 50% is improved by 15% to 35%. The risk level above 80% dropped from 54% to 33%. The experimental results demonstrate our placement strategies can greatly improve the security level of the entire cloud.

Fig. 10.

Comparison with distribution in 1600 VMs and 120 physical machines.

Fig. 11.

Accumulative risk value.

5.6. Compatible with other virtual machine allocation policy

The nature of our algorithm makes it more suitable to the VMP initialization and re-optimization procedure. When a VM is deployed or re-activated, the new placement will be generated based on the current one to achieve the multi-objective optimization, while keeping the low migration cost in mind. To suit the request, we modify our original SMOOP algorithm. As presented in Algorithm 4, the crossover operation is removed and the mutation operation is modified to adapt the latest VM allocation policy. The mutation operation will always operate based on the current placement, in order to limit the number of migrated VMs.

Algorithm 4

SMOOP-phase2

Also, we considered two currently existing VM allocation policies in the mutation operation to defeat the co-residence attack. The CLR (Co-located Resistant) could be easily implemented into our mutation operation without major modification. The mutation operation is revised as Algorithm 5.

Algorithm 5

Mutation(X)

In our previous work, we did not consider VMs with their associated user account. All VMs were treated as the same. This setting simplifies the modeling, and it well fits for the situation of not-knowing the advisor’s account. However, it makes PSSF (Previous-Selected-Server-First) hard to be adapted. To adapt PSSF, two preliminary settings should be supported. The VM deployment history for a particular user should be maintained and all VMs need to be grouped by their owner’s account. Afterwards, PSSF can be integrated as other VM allocation policies in our mutation operation.

6. Conclusion

In this paper, we have studied the comprehensive security assessment of virtual machine placement strategies and presented an approach to comprehensively quantify the security risks of the cloud based on the vulnerabilities caused by various factors, including the network, the physical machines, the VMs, and the co-residence of VMs. To optimize these objectives, we have designed a new scheme to generate VMP based on multiple objective optimization with the given resources and other constraints. Our proposed strategy seeks the Pareto-optimal placement while considering multiple optimization objectives and constraints. The experimental results demonstrate the effectiveness of our approach and the improvement compared to existing solutions. On the other hand, our current experiments are simulation based. In the future, we plan to continue our study and experiment in a real-world environment. As we discussed, the priority of strategies can be adjusted in real-time and new strategy might need to be added, in response to the runtime environment. In this way, our system could keep evolving to fit real-world situations.

Footnotes

Acknowledgment

This project is partially supported by ARO grant W911NF-15-1-026 and NSF grants under CNS-163441 and CNS-1524462.

References

Afoulki,

Bousquet and

Rouzaud-Cornabas, A security-aware scheduler for virtual machines on iaas clouds, Report, 2011.

Al-Haj,

Al-Shaer and

H.V.

Ramasamy, Security-aware resource allocation in clouds, in: 2013 IEEE International Conference on Services Computing, 2013, pp. 400–407. doi:10.1109/SCC.2013.36.

Alicherry and

T.V.

Lakshman, Optimizing data access latencies in cloud systems by intelligent virtual machine placement, in: 2013 Proceedings IEEE INFOCOM, 2013, pp. 647–655, ISSN 0743-166X. doi:10.1109/INFCOM.2013.6566850.

Amazon, Amazon Web Services. http://aws.amazon.com.

Aviram,

Hu,

Ford and

Gummadi, Determinating timing channels in compute clouds, in: Proceedings of the 2010 ACM Workshop on Cloud Computing Security Workshop, CCSW ’10, ACM, New York, NY, USA, 2010, pp. 103–108. ISBN 978-1-4503-0089-6. doi:10.1145/1866835.1866854. http://doi.acm.org/10.1145/1866835.1866854.

Azar,

Kamara,

Menache,

Raykova and

Shepard, Co-location-resistant clouds, in: Proceedings of the 6th Edition of the ACM Workshop on Cloud Computing Security, CCSW ’14, ACM, New York, NY, USA, 2014, pp. 9–20. ISBN 978-1-4503-3239-2. doi:10.1145/2664168.2664179. http://doi.acm.org/10.1145/2664168.2664179.

Barham,

Dragovic,

Fraser,

Hand,

Harris,

Ho,

Neugebauer,

Pratt and

Warfield, Xen and the art of virtualization, SIGOPS Oper. Syst. Rev. 37(5) (2003), 164–177, ISSN 0163-5980. doi:10.1145/1165389.945462. http://doi.acm.org/10.1145/1165389.945462.

S.A.

Baset,

Wang and

Tang, Towards an understanding of oversubscription in cloud, in: Proceedings of the 2nd USENIX Conference on Hot Topics in Management of Internet, Cloud, and Enterprise Networks and Services, Hot-ICE ’12, USENIX Association, Berkeley, CA, USA, 2012, p. 7. http://dl.acm.org/citation.cfm?id=2228283.2228293.

Bin,

Biran,

Boni,

Hadad,

E.K.

Kolodner,

Moatti and

D.H.

Lorenz, Guaranteeing high availability goals for virtual machine placement, in: 2011 31st International Conference on Distributed Computing Systems, 2011, pp. 700–709, ISSN 1063-6927. doi:10.1109/ICDCS.2011.72.

10.

Bistarelli,

S.N.

Foley and

O’Sullivan, Detecting and eliminating the cascade vulnerability problem from multilevel security networks using soft constraints, in: Proceedings of the 16th Conference on Innovative Applications of Artifical Intelligence, IAAI ’04, AAAI Press, 2004, pp. 808–813. ISBN 0-262-51183-5. http://dl.acm.org/citation.cfm?id=1597321.1597329.

11.

R.N.

Calheiros,

Ranjan,

Beloglazov,

C.A.F.

De Rose and

Buyya, CloudSim: A toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms, Software: Practice and Experience 41(1) (2011), 23–50, ISSN 1097-024X. doi:10.1002/spe.995. https://dx-doi-org.web.bisu.edu.cn/10.1002/spe.995.

12.

Caron,

A.D.

Le,

Lefray and

Toinard, Definition of security metrics for the cloud computing and security-aware virtual machine placement algorithms, in: 2013 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, 2013, pp. 125–131. doi:10.1109/CyberC.2013.28.

13.

CloudSim, http://www.cloudbus.org/cloudsim/.

14.

CVE-2007-4993, Xen guest root can escape to domain 0 through pygrub. http://cve.mitre.org/cgibin/cvename.cgi?name=CVE-2007-4993.

15.

CVE-2007-5497, Vulnerability in xenserver could result in privilege escalation and arbitrary code execution. http://suuport.citrix.com/article/CTX118766.

16.

Deb,

Pratap,

Agarwal and

Meyarivan, A fast and elitist multiobjective genetic algorithm: NSGA-II, IEEE Transactions on Evolutionary Computation 6(2) (2002), 182–198. doi:10.1109/4235.996017.

17.

Dósa and

Sgall, First fit bin packing: A tight analysis, in: 30th International Symposium on Theoretical Aspects of Computer Science (STACS 2013),

Portier and

Wilke, eds, Leibniz International Proceedings in Informatics (LIPIcs), Vol. 20, Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 2013, pp. 538–549, ISSN 1868-8969. ISBN 978-3-939897-50-7. doi:10.4230/LIPIcs.STACS.2013.538. http://drops.dagstuhl.de/opus/volltexte/2013/3963.

18.

Dósa and

Sgall, Optimal analysis of best fit bin packing, in: Automata, Languages, and Programming: 41st International Colloquium, ICALP 2014, Copenhagen, Denmark, July 8–11, 2014, Proceedings, Part I,

Esparza,

Fraigniaud,

Husfeldt and

Koutsoupias, eds, Springer, Berlin, Heidelberg, 2014, pp. 429–441. doi:10.1007/978-3-662-43948-7_36. https://doi.org/10.1007/978-3-662-43948-7_36.

19.

Hacker, Xbox 360 Hypervisor Privilege Escalation Vulnerability. http://www.securityfocus.com/archive/1/461489.

20.

Han,

Chan,

Alpcan and

Leckie, Virtual machine allocation policies against co-resident attacks in cloud computing, in: 2014 IEEE International Conference on Communications (ICC), 2014, pp. 786–792, ISSN 1550-3607. doi:10.1109/ICC.2014.6883415.

21.

Han,

Chan,

Alpcan and

Leckie, Using virtual machine allocation policies to defend against co-resident attacks in cloud computing, IEEE Transactions on Dependable and Secure Computing 14(1) (2017), 95–108, ISSN 1545-5971. doi:10.1109/TDSC.2015.2429132.

22.

J.D.

Horton,

Cooper,

Hyslop,

B.G.

Nickerson,

Ward,

Harland,

Ashby and

Stewart, The cascade vulnerability problem, Journal of Computer Security 2(4) (1993), 279–290. doi:10.3233/JCS-1993-2402.

23.

Jansen and

P.R.

Brenner, Energy efficient virtual machine allocation in the cloud, in: 2011 International Green Computing Conference and Workshops, 2011, pp. 1–8. doi:10.1109/IGCC.2011.6008550.

24.

Jhawar,

Piuri and

Samarati, Supporting security requirements for resource management in cloud computing, in: 2012 IEEE 15th International Conference on Computational Science and Engineering, 2012, pp. 170–177. doi:10.1109/ICCSE.2012.32.

25.

R.E.

Korf, A new algorithm for optimal bin packing, in: Eighteenth National Conference on Artificial Intelligence, American Association for Artificial Intelligence, Menlo Park, CA, USA, 2002, pp. 731–736. ISBN 0-262-51129-0. http://dl.acm.org/citation.cfm?id=777092.777205.

26.

KVM, Kernel Based Virtual Machine. http://www.linux-kvm.org.

27.

Li,

Zhang,

Bai,

Zang,

Yu and

He, Improving Cloud Survivability through Dependency based Virtual Machine Placement, 2012, pp. 321–326.

28.

Z.A.

Mann, Allocation of virtual machines in cloud data centers – A survey of problem models and optimization algorithms, ACM Comput. Surv. 48(1) (2015), 11–11134, ISSN 0360-0300. doi:10.1145/2797211. http://doi.acm.org/10.1145/2797211.

29.

Maziku and

Shetty, Network aware VM migration in cloud data centers, in: 2014 Third GENI Research and Educational Experiment Workshop, 2014, pp. 25–28. doi:10.1109/GREE.2014.18.

30.

D.H.

Phan,

Suzuki,

Carroll,

Balasubramaniam,

Donnelly and

Botvich, Evolutionary multiobjective optimization for green clouds, in: Proceedings of the 14th Annual Conference Companion on Genetic and Evolutionary Computation, GECCO ’12, ACM, New York, NY, USA, 2012, pp. 19–26. ISBN 978-1-4503-1178-6.

31.

Ristenpart,

Tromer,

Shacham and

Savage, Hey, you, get off of my cloud: Exploring information leakage in third-party compute clouds, in: Proceedings of the 16th ACM Conference on Computer and Communications Security, CCS ’09, ACM, New York, NY, USA, 2009, pp. 199–212. ISBN 978-1-60558-894-0. doi:10.1145/1653662.1653687. http://doi.acm.org/10.1145/1653662.1653687.

32.

Rutkowska and

Wojtczuk, Xen Owning Trilogy, Talk at Black Hat (2008).

33.

Servin,

Ceberio,

Freudenthal and

Bistarelli, An optimization approach using soft constraints for the cascade vulnerability problem, in: NAFIPS 2007 – 2007 Annual Meeting of the North American Fuzzy Information Processing Society, 2007, pp. 372–377. doi:10.1109/NAFIPS.2007.383867.

34.

Shetty,

Yuchi and

Song, Security-aware virtual machine placement in cloud data center, in: Moving Target Defense for Distributed Systems, Springer, 2016, pp. 13–24. doi:10.1007/978-3-319-31032-9_2.

35.

Shi,

Song,

Chen and

Zang, Limiting cache-based side-channel in multi-tenant cloud using dynamic page coloring, in: 2011 IEEE/IFIP 41st International Conference on Dependable Systems and Networks Workshops (DSN-W), 2011, pp. 194–199, ISSN 2325-6648. doi:10.1109/DSNW.2011.5958812.

36.

Shigeta,

Yamashima,

Doi,

Kawai and

Fukui, Design and implementation of a multi-objective optimization mechanism for virtual machine placement in cloud computing data center, in: International Conference on Cloud Computing, Springer, 2012, pp. 21–31.

37.

Varadarajan,

Zhang,

Ristenpart and

M.M.

Swift, A placement vulnerability study in multi-tenant public clouds, in: USENIX Security, 2015, pp. 913–928.

38.

Wu,

Ding,

Lin,

Min-Allah and

Wang, XenPump: A new method to mitigate timing channel in cloud computing, in: 2012 IEEE Fifth International Conference on Cloud Computing, 2012, pp. 678–685, ISSN 2159-6182. doi:10.1109/CLOUD.2012.28.

39.

Xu and

J.A.B.

Fortes, Multi-objective virtual machine placement in virtualized data center environments, in: Green Computing and Communications (GreenCom), 2010 IEEE/ACM Int’l Conference on Int’l Conference on Cyber, Physical and Social Computing (CPSCom), 2010, pp. 179–188. doi:10.1109/GreenCom-CPSCom.2010.137.

40.

Xu,

Cui and

Peinado, Controlled-channel attacks: Deterministic side channels for untrusted operating systems, in: 2015 IEEE Symposium on Security and Privacy, 2015, pp. 640–656, ISSN 1081-6011. doi:10.1109/SP.2015.45.

41.

Xu,

Wang and

Wu, A measurement study on co-residence threat inside the cloud, in: 24th USENIX Security Symposium (USENIX Security 15), USENIX Association, Washington, D.C., 2015, pp. 929–944. ISBN 978-1-931971-232. https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/xu.

42.

Yu,

Gui,

Tian,

Yang and

Zhao, A security-awareness virtual machine placement scheme in the cloud, in: 2013 IEEE 10th International Conference on High Performance Computing and Communications 2013 IEEE International Conference on Embedded and Ubiquitous Computing, 2013, pp. 1078–1083. doi:10.1109/HPCC.and.EUC.2013.152.

43.

Yuchi and

Shetty, Enabling security-aware virtual machine placement in IaaS clouds, in: MILCOM 2015 – 2015 IEEE Military Communications Conference, 2015, pp. 1554–1559. doi:10.1109/MILCOM.2015.7357666.

44.

Zhang,

Jia,

Wang,

Zhang,

Huang,

Wang and

Liu, A comprehensive study of co-residence threat in multi-tenant public PaaS clouds, in: Information and Communications Security, Springer, 2016, pp. 361–375. doi:10.1007/978-3-319-50011-9_28.

45.

Zhang,

Juels,

M.K.

Reiter and

Ristenpart, Cross-tenant side-channel attacks in PaaS clouds, in: Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security, CCS ’14, ACM, New York, NY, USA, 2014, pp. 990–1003. ISBN 978-1-4503-2957-6.

46.

Zhang,

Li,

Bai,

Yu and

Zang, Incentive compatible moving target defense against VM-colocation attacks in clouds, in: Information Security and Privacy Research: 27th IFIP TC 11 Information Security and Privacy Conference, Section 2012, Heraklion, Crete, Greece, June 4–6, 2012, Proceedings,

Gritzalis,

Furnell and

Theoharidou, eds, Springer, Berlin, Heidelberg, 2012, pp. 388–399. doi:10.1007/978-3-642-30436-1_32.