Research on comprehensive virtualization performance evaluation method

Abstract

With the development and popularization of virtualization technology, more and more enterprises tend to deploy applications to virtual machines, in order to reduce cost and improve the utilization rate of resources in the meantime. On the basis of the combination method of the comprehensive performance assessment of virtualization, this article aim to find a way to measure the performance percentage of the virtual machine against the physical machine percentage as a whole, which is built on the method of Fuzzy Analytic Hierarchy Process and the principle of maximizing deviations. According to the shortcoming of the Analytic Hierarchy Process, this article presents a way to calculate weight of the functional layer elements based on the method of Fuzzy Analytic Hierarchy Process. The experimental results show that the conclusions drawn from the comprehensive performance evaluation method for virtualization are consistent with the experimental results.

Keywords

Virtual machine FAHP maximizing deviation decision-making method performance test LoadRunner

1 Introduction

Virtualization has rapidly attained mainstream status in different fields by delivering transformative cost savings as well as increased operational efficiency and flexibility. The virtual machines allow the computer resources to be partitioned for multiple isolated user space instances, so that it can save a lot of resources meanwhile just like physical machines [1]. Therefore, the virtualization technology is introduced into the field of software testing to reduce thecost [2].

Virtual machine is achieved by means of hypervisor, also known as Virtual Machine Monitors (VMMs), which will results in a certain performance hit compared with the physical machine. In the decision making progress of enterprise virtualization, how to evaluate the performance percentage of the virtual machines over their physical counterparts play a key role. However, the current research on the comparison between the performance of virtual machine and physical machine is based on the performance comparison of a single dimension, such as the comparison of the CPU, the network, the memory or the disk. Hence the comprehensive performance assessment about the virtual machine and the physical machine needs to be further researched in the future.

The VMMs play a vital role on the performance of the whole system, meanwhile differences between different VMMs exist. For the purpose of conducting a quantitative assessment on the performance loss of virtual machines over their physical counterparts with the same configuration, and solving the problem of quantitative evaluation of performance differences between virtual products, a method is presented in this paper to assessing the performance losses and differences quantitatively.

The main work of this paper is as follows:

The Fuzzy Analytic Hierarchy Process (Fuzzy-AHP) method has been applied in comprehensive performance evaluation, in order to improve the Analytic Hierarchy Process (AHP) which is used in traditional decision-making method.

A new method has been proposed to calculate the weight of the indicator layer element based on the principle of maximizing deviations. The proposed method simplified the calculation process, and enhancing the original maximizing deviation method.

The process of combining two calculation methods with the actual comprehensive performance evaluation method was described.

The rest of the paper is organized as follows: A brief overview of the related work is introduced in Section 2. In Section 3, an enhanced method to calculate the indicator layer element weight based on the principle of maximizing deviations is presented. Simulation experiments are performed in Section 4 and final conclusion is given in Section 5.

2 Related works

Many studies have explored the field of virtualization performance test. At present, most of them are focused on: using virtual products as test tools, doing functional and performance test on virtual products, comparing the performance of individual module between different virtual products. The issue is the research of virtualization oriented comprehensive performance testing is relatively less, especially the research of the quantitative performance evaluation test.

The enterprise application systems nowadays need to interact with many other heterogeneous systems. This distributed, large-scale, heterogeneous environment makes it difficult to measure the performance and scalability of the system. Schneider [3] and others generated a colored petri net with the help of virtualization technology to test the property of enterprise complex system. While Kim Y [4] et al. applied virtualization technology to the test of the model driven distributed system. To save the hardware and software resources, Lazić [5] made use of virtualization technology at first, and then they combined virtualization and combinatorial test in order to reduce the number of generated test cases and ensure the accuracy and reliability of the software under test. Thus it can be seen that when testing software, the assist of virtualization contributes much in promoting efficiency and cost savings.

The performance of two open source virtual monitor Xen and KVM was compared by Ben-Yehuda [11] and etc. They measured the compile performance of CPU core and the I/O performance of IOZone. The result shows that, KVM has less CPU performance but better I/O performance compare to Xen. However, the contrast is single-dimensional because of it simply compared the single function of physical and virtual machine. Benevenuto [12] used some common hardware and software configuration parameters to build models to predict the performance of applications deployed in Xen, but the configuration parameters of different application are not the same, the model can’t directly extended to other applications. Chaudhary et al. [13] used standard benchmarks to compare the different classes of VMM, including network deployment, SMP performance, file system and MPI availability. However, the focus of the test was on the comparison of virtual machine usage characteristics under high performance (HPC) computing conditions.

Studies above show that, ways to compare the comprehensive performance between physical machines and virtual machines should be more deeply researched. A layered virtualization oriented comprehensive performance evaluation method in this paper is presented to evaluate the comprehensive performance of physical machines and virtual machines, which would offer theoretical basis for enterprise virtualization.

3 Methodology

The approach proposed in the paper combined Fuzzy-AHP with the enhanced maximizing deviations method to evaluate the comprehensive performance of physical machines and virtual machines. The former will be utilized to calculate the weight of functional layer element while the latter to calculate that of indicator layer element. This section will describe the detailed derivation process of the method below.

3.1 Fuzzy Analytic Hierarchy Process (Fuzzy-AHP)

The Fuzzy-AHP is a fuzzy prioritization method, which is an improvement ground on the traditional Analytic Hierarchy Process (AHP). It is a multi-attribute decision-making method which is based on fuzzy logic concepts [9].

In the paper, the fuzzy-AHP making deal with the problem of calculating the functional layer weight. There are three main steps as follows in the computing process.

Step 1: Construct a fuzzy judgment matrix. $R = [\begin{matrix} r_{11} & r_{12} & \dots & r_{1 n} \\ r_{21} & r_{22} & \dots & r_{2 n} \\ ⋮ & ⋮ & ⋮ \\ r_{n 1} & r_{n 2} & \dots & r_{nn} \end{matrix}], i, j = 1, 2, \dots, n, r_{ij} \in [0, 1]$ (3-1-1)

Step 2: Regulate the consistency of Fuzzy Judgment Matrix.

Step 3: Calculate the weight by Least Squares Method. $w_{i} = \frac{1}{n} - \frac{1}{2 a} + \frac{1}{n α} \sum_{k = 1}^{n} r_{ik}, i = 1, 2, \dots, n .$ (3-1-2)

Where the linear coefficient α satisfies $α \geq \frac{n - 1}{2}$ . While the concrete weight is closely related to α and the layer of weight will be clearer when α is smaller, we use $α = \frac{n - 1}{2}$ in the approach.

3.2 The enhanced maximizing deviation method

The article [8] presented a general algorithm based on maximizing deviation method which ignored the dominance of the functional layer to the indicator layer element. An improved method was put forward in the article [10], however, the constraint of the method must satisfy the constraint condition of weight unitization which means that the quadratic sum of weight must be 1, which does not accord with the real situation that the weight sumsup to 1.

In this paper, an enhanced method is presented to calculate the weight of indicator layer element. The method has not only taken the dominance of the functional layer to the indicator layer element into account, but also avoided the constraint of weight unitization which is necessary in the method of maximizing deviation decision-making. The concrete process is described below:

Step 1: Let P_j represent an arbitrary indicator element, and the weight of the functional layer element it belonged to is $w_{j}^{'}$ . Then the deviation of attribute P_j in indicator layer scheme G_i (have considered about the weight of functional layer) and the same attribute in other scheme is: $Z_{ij}^{*} = \sum_{k = 1}^{m} | (y_{ij} - y_{kj}) w_{j}^{'} |$ (3-2-1)

Step 2: The deviation of attribute P_j in other scheme is: $\sum_{i = 1}^{m} Z_{ij}^{*} = \sum_{i = 1}^{m} \sum_{k = 1}^{m} | (y_{ij} - y_{kj}) w_{j}^{'} |$ (3-2-2)

Step 3: Therefore, the total deviation of all attribute is: $Z^{*} = \sum_{j = 1}^{n} \sum_{i = 1}^{m} Z_{ij}^{*} = \sum_{j = 1}^{n} \sum_{i = 1}^{m} \sum_{k = 1}^{m} | (y_{ij} - y_{kj}) w_{j}^{'} |$ (3-2-3)

Step 4: According to the principle of maximum deviation, the deviation of attribute P_j is larger, the corresponding weight is larger. In this condition, $w_{j}^{*}$ (the weight of attribute) is: $w_{j}^{*} = \frac{\sum_{i = 1}^{m} Z_{ij}^{*}}{\sum_{j = 1}^{n} \sum_{i = 1}^{m} Z_{ij}^{*}} = \frac{\sum_{i = 1}^{m} \sum_{k = 1}^{m} | (y_{ij} - y_{kj}) w_{j}^{'} |}{\sum_{j = 1}^{n} \sum_{i = 1}^{m} \sum_{k = 1}^{m} | (y_{ij} - y_{kj}) w_{j}^{'} |}$ (3-2-4)

The result of this method is as same as the original maximum deviation method, furthermore, it satisfies the constraint condition of weight unitization (the quadratic sum of weight must be 1) and the actual weight sums up to 1. In addition, the method is simpler.

3.3 The model of comprehensive performance evaluation

From Fig. 1, the model adopts Fuzzy-AHP to calculate the weight of functional layer, and the improved maximizing deviation method to process the indicator layer.

Fig.1

The model of three layer performance evaluation.

3.3.1 The calculation model of functional layer

In the process of evaluate the performance of physical machines and virtual machines with same configuration, their usage should be confirmed firstly. For instance, since computational server used for scientific computation depends more on CPU property, the evaluation of the CPU plays a more important role than other functional elements during the wholeprocess. The basis of building fuzzy matrix is to diversify the importance of each functional element according to their different usages.

Step 1: Construct the performance evaluation model for functional layer.

The model divides the overall goal of the performance evaluation into four parts which are CPU, memory, network and I/O, as Fig. 2:

Fig.2

The elements of functional layer.

Step 2: Construct a 4-dimensional (n = 4) fuzzy judgment matrix R (r_ij) _4×4 in accordance with the usage of system.

Step 3: Regulate the consistency of fuzzy judgment matrix when the consistency is dissatisfied.

Step 4: According to the linear relation between the fuzzy judgment matrix element and weight: r_ij = 0.5 + α (w_i - w_j), the weight of elements w_i, i = 1, 2, 3, 4 can be determined by Least Squares Method: $w_{i} = \frac{1}{4} - \frac{1}{2 a} + \frac{1}{4 α} \sum_{k = 1}^{4} r_{ik}, i = 1, 2, \dots, 4$ (3-3-1)

Where α = ³/₂, hence the weight is $w_{i} = \frac{1}{6} \sum_{k = 1}^{4} r_{ik} - \frac{1}{12}, i = 1, 2, \dots, 4$ (3-3-2)

Thereafter, the functional layer weight can be calculated with the value in fuzzy judgment matrix.

3.3.2 The calculation model of indicator layer

After calculating the weight of functional layer, the next step is to decompose it into specific and measurable indicators. By means of which, the performance of functional elements can be further evaluated. The process is described as below:

CPU measurement indicators as Table 1

Memory measurement indicators as Table 2

I/O measurement indicators as Table 3

Net measurement indicators as Table 4

Table 1
CPU measurement indicators

Measurement Indicators Significance

Integer index Scientific Computing for integer

Float-point index Scientific Computing for float and double

Measurement Indicators	Significance
Integer index	Scientific Computing for integer
Float-point index	Scientific Computing for float and double

Table 2

Memory measurement indicators

Measurement Indicators	Significance
memory write	Write data to the memory
pipe communication	Communicate by memory pipe
memory mapping latency	The time consuming in memory mapping
memory mapping repeated read	Read memory mapping repeatedly
Memory copy	Copy the data of memory
Open/close files	Close the file after open it immediately

Table 3

I/O measurement indicators

Measurement Indicators	Significance
Sequential read	Read files in sequential order
Sequential write	Write files in sequential order
Random read	Read files in random order
Random write	Write files in random order
Repeatable read	Read files repeatedly
Repeatable write	write files repeatedly
Backwards Read	Read files in reverse order
Stride Read	Stride read

Table 4

Net measurement indicators

Measurement Indicators	Significance
TCP File Transfer	Network communication by TCP protocol
UDP File Transfer	Network communication by UDP protocol

By decomposing the functional layer, the decision process get 18 indicator elements (n = 18). Scheme set G has two elements (m = 2) as it represent the decision about physical machine and virtual machine. Thereafter, the weight of indicator layer element can be calculated using the method proposed in 3.2. The process is described as below:

Step 1: Measure the performance of each indicator layer elements by benchmark test programs to get the performance parameter of single indicator. The attribute value of an indicator layer element is the element’s performance ratio of virtual machines to physical machine. This should be processed by dimensionless method.

Step 2: Calculate the deviation of every indicator elements using expression (3-2-2). $\sum_{i = 1}^{2} Z_{ij}^{*} = \sum_{i = 1}^{2} \sum_{k = 1}^{2} | (y_{ij} - y_{kj}) w_{j}^{'} |, (j = 1, 2, \dots, 18)$ (3-3-3)

Step 3: Calculate the total deviation of all the indicator elements using expression (3-2-3). $Z^{*} = \sum_{j = 1}^{18} \sum_{i = 1}^{2} Z_{ij}^{*} = \sum_{j = 1}^{18} \sum_{i = 1}^{2} \sum_{k = 1}^{2} | (y_{ij} - y_{kj}) w_{j}^{'} |$ (3-3-4)

Step 4: Finally, the weight of indication elements which is described as $w_{j}^{*}$ can be obtained.

$\begin{matrix} w_{j}^{*} = \frac{\sum_{i = 1}^{2} Z_{ij}^{*}}{\sum_{j = 1}^{18} \sum_{i = 1}^{2} Z_{ij}^{*}} = \frac{\sum_{i = 1}^{2} \sum_{k = 1}^{2} | (y_{ij} - y_{kj}) w_{j}^{'} |}{\sum_{j = 1}^{18} \sum_{i = 1}^{2} \sum_{k = 1}^{2} | (y_{ij} - y_{kj}) w_{j}^{'} |}, \\ j = 1, 2, \dots, 18 \end{matrix}$ (3-3-5)

3.3.3 Calculate total score

Specific to concrete application situation, firstly calculate the functional elements weight using expression (3-3-2) which is derived by Fuzzy-AHP method, after that use expression (3-3-5) that obtained by the enhanced maximizing deviation method proposed in the article to compute the weight of indicator elements ( $w_{j}^{*}$ , j = 1, 2, ⋯, n), finally we can get a total score of scheme G_i: $T_{i} = \sum_{j = 1}^{n} w_{j}^{*} p_{ij}$ (3-3-6)

N employee 18 in this article as the number of indicator layer elements is 18.

4 Simulation and results analysis

To obtain the percentage of performance loss of virtual machine in VMM layer, this paper applies the comprehensive performance evaluation method into comparing the overall performance between physical machine and virtual machine with the same configuration. After that, a series of performance experiments will be conducted to verify the result we got in the previous step. Detailed results for each of the experiments would be provided to makea comparison.

4.1 Calculate the performance loss percentage of virtual machine

The virtualization environment was built by VMware Workstation (Vw11), the host operating system and the client operating system is Ubuntu12.04 LTS. Considering about the performance of image servers have a great influence on user experience when it is deployed on the virtualization environment, we built the scene based on the image transfer/access of server. The data flow of image server is as Fig. 3:

Fig.3

Image server data flow.

The whole process shows that the data flow in the image servers is completed by the cooperation of CPU, memory, I/O and network, the importance degree of them are ranked as: network>I/O>memory>CPU.

4.1.1 Calculate the weight of functional layer

Step 1: Construct Table 5 with the method of Fuzzy-AHP.

Step 2: The fuzzy consistent judgment matrix is: $R = (r_{ij})_{4 \times 4} = [\begin{matrix} 0.5 0.4 0.7 0.6 \\ 0.6 0.5 0.8 0.7 \\ 0.3 0.2 0.5 0.4 \\ 0.4 0.3 0.6 0.5 \end{matrix}]$

Step 3: According to expression (3-1-2), the weight of functional layer is showed in Table 6:

Table 5
The fuzzy judgment table of functional layer

IO Network CPU Memory

IO 0.5 0.4 0.7 0.6

Network 0.6 0.5 0.8 0.7

CPU 0.3 0.2 0.5 0.4

Memory 0.4 0.3 0.6 0.5

	IO	Network	CPU	Memory
IO	0.5	0.4	0.7	0.6
Network	0.6	0.5	0.8	0.7
CPU	0.3	0.2	0.5	0.4
Memory	0.4	0.3	0.6	0.5

Table 6

The weight of functional layer

elements	IO	Network	CPU	Memory
weight	0.28	0.35	0.15	0.22

4.1.2 Calculate the weight of indicator layer

The indicators of functional layer introduced above require different measuring tools to evaluate the performance. Four tools was introduced in the article to measure the performance of CPU, memory, I/O and network, they are NBench, LMBench, IOZone and NetIO+. Certainly, in order to adapting to the test scenario, the tools are improved to a certainextent.

After several tests, we got performance data of CPU, memory, I/O and network. By processing these data with dimensionless method, the performance comparison data of indicator layer elements between physical and virtual machines can be calculated. The results are as follows (from Tables 7–10).

Table 7
CPU performance comparison

INTEGER FLOATING-POINT

INDEX INDEX

Physical machine 1 1

Virtual machine 0.9896 0.9759

	INTEGER	FLOATING-POINT
Physical machine	1	1
Virtual machine	0.9896	0.9759

Table 8

Memory performance comparison

	Mem	Pipe	Bcopy	Mmap	Open
	Write			Latency	Close
Physical machine	1	1	1	1	1
Virtual machine	0.6341	0.9048	0.8637	0.8976	0.8304

Table 9

Network performance comparison

	TCP	UDP
Physical machine	1	1
Virtual machine	0.9757	0.8418

Table 10

IO performance comparison

	Read	Write	R-read	R-write	Backwards Read	Random Read	Random Write	Strided Read
Physical machine	1	1	1	1	1	1	1	1
Virtual machine	0.8730	0.8949	0.8867	0.9420	0.8872	0.8786	0.8958	0.8811

According to expressions (3-3-3)(3-3-4) and the data above, the sum of deviation of indicator layer elements is Z = Z1 + Z2 + Z3 + Z4 = 1.0483484815813. Z1, Z2, Z3 and Z4 are the sum of deviation of each functional layer elements separately.

At last we need to calculate the weight of each indicator layer elements by using the expression (3-3-5).

eg. The weight of INTEGER INDEX $w_{1}^{*} = \frac{0.15 * 0.0104 * 2}{1.0483484815813} = 0.00297611 .$

4.1.3 Calculate total score

N employee 18 in this article as the number of indicator layer elements is 18.

4.2 Verify the percentage by concrete experiment

The performance comparison percentage between physical and virtual machine is calculated in the application of image server. Therefore, the concrete experiment should be under the same practical background. Considering about that the image server is an integral part of the e-commerce website, we use Ecmall (an open source e-commerce website which is based on PHP and MySQL) to do the final experiment. The LoadRunner which can simulate the situation that many users are checking about product details could be used to test the performance of Ecmall.

In the experiment, Ecmall was deployed in physi-cal machine and virtual machine separately. After that, The LoadRunner made 4 group experiments to do pressure test in order to measure the performance of Ecmall.

According to the results of 4 group experiments, the percentage of performance of the virtual machine compared to physical machine are as Table 11.

Table 11
The percentage of performance

PerfVu2 PerfVu4 PerfVu6 PerfVu8

The percentage of performance 86.17% 83.21% 84.07% 86.06%

	PerfVu2	PerfVu4	PerfVu6	PerfVu8
The percentage of performance	86.17%	83.21%	84.07%	86.06%

The percentage of performance calculated by Fuzzy-AHP and improved method based on maximizing deviations is 84.27%. The average error between the results showed by Tables 3–4 and the result calculated by the method is smaller than 2%. We can conclude that the result of experiment conforms to the calculation.

5 Conclusion and future work

In this paper, the method of fuzzy Analytic Hierarchy Process (AHP) and weighting method of index layer element based on the idea of maximizing deviation are used to design a comprehensive performance evaluation method for virtualization. This can effectively evaluate the overall performance of the virtual machine and provide the decision-making plan for the enterprise’s virtual deployment. Then, the proposed method was tested by deploying virtualization environment which is constructed by image server application. The performance percentage value obtained by the experiment could play a certain role in the enterprise virtualization decision making process. At last the method is practiced in an application environment and verified by multi-group experiments with the LoadRunner testing tool. In this way, it has proved that the conclusion drawn from the method of comprehensive performance assessment of virtualization is consistent with the experimentalresult.

Finally, the future work can be extended by combining different benchmark programs in the paper to bring convenience to the users. Assume that the graphical user interface was built, the evaluation method will be more flexible and intuitive.

Footnotes

Acknowledgments

This research was supported by the National Nature Science Foundation of China (No. 61370103), Guangdong Province Science & Technology Fund (2014B010103002), Guangzhou Produce & Research Fund (201508010057) and the Open Fund of State Key Laboratory of Wuhan University Software Engineering.

References

Kusic

, Kephart

J.O.

, Hanson

J.E.

, et al. Power and performance management of virtualized computing environments via lookahead control[J], International Conference on Autonomic Computing12(1) (2008), 3–12.

Weltzin

and Delgado

, Using virtualization to reduce the cost of test[C]. Autotestcon, IEEE. IEEE, 2009, pp. 439–442.

, Han

, Schneider

J.G.

, et al. A virtual deployment testing environment for enterprise software systems[C], Proceedings of the 8th international ACM SIGSOFT conference on Quality of Software Architectures ACM, 2012, pp. 101–110.

Kim

, Lee

and Kim

, Virtualization for testing in model-driven distributed system[J], IEEE Vehicular Technology Conference413(4) (2012), 1–5.

Lazić

, Popovic

and Mastorakis

, A simultaneous application of combinatorial testing and virtualization as a method for software testing[J], WSEAS Transactions on Information Science and Applications2009(11), 1802–1813.

Kim

, Moon

, Song

, et al. Software Performance Testing Scheme Using Virtualization Technology[C], International Conference on Ubiquitous Information Technologies & Applications IEEE, 2009, pp. 1–5.

Martignoni

, Paleari

, Fresi Roglia

, et al. Testing system virtual machines[C], Proceedings of the 19th International Symposium on Software Testing and Analysis ACM, 2010, pp. 171–182.

Wang

, Using the method of maximizing deviations to make decision for multiindicies [J], Systems Engineering and Electronics (7) (1998), 24–26.

Zhang

, Fuzzy analytical hierarchy process (FAHP)[J], Fuzzy Systems and Mathematics14(2) (2000), 80–88.

10.

Peng

, Zhang

, Lie

, et al. The improved decision-make module of maximizing deviations and its application in multi-task planning of near space[J], System Engineering Theory and Practice34(2) (2014), 421–427.

11.

Deshane

, Shepherd

, Matthews

, et al. Quantitative comparison of Xen and KVM[J]. Xen Summit, Boston, MA, USA, 2008, pp. 1–2.

12.

Benevenuto

, Fernandes

, Santos

, et al. Performance Models for Virtualized Applications[J], Frontiers of High Performance Computing & Networking – Ispa Workshops, 2006, pp. 427–439.

13.

Chaudhary

, Cha

, Walters

J.P.

, et al. A comparison of virtualization technologies for HPC[C], Advanced Information Networking and Applications, 2008 AINA 2008 22nd International Conference on IEEE, 2008, pp. 861–868.

	INTEGER	FLOATING-POINT
	INDEX	INDEX
Physical machine	1	1
Virtual machine	0.9896	0.9759