October 2005 Page 1

LAMP Performance Characterization and Sizing on VMware ESX Server® 2.5.1 and Dell™ PowerEdge™ 1855 Blade

Servers

By Amresh Singh, J. Craig Lowery, Ph.D., C S Prasanna Nanda, Rudramuni B, and Scott Stanford

October 2005 Page 2

This paper provides the findings from performance characterization and sizing tests for Dell PowerEdge 1855 Blade Server running VMware ESX Server 2.5.1. Dell Scalable Enterprise team ran tests that simulated an OLTP (Online Transaction Processing) workload in a virtualized LAMP (Linux®, Apache, MySQL, and PHP) environment. By using the analysis of the test results in this article, Systems Engineers and IT Architects can determine the appropriate number of VMs to host on top of ESX Server for a typical transactional LAMP based workload; how to optimize the configuration for VMs (Virtual Machines); and how to achieve effective workload and virtual machine scaling.

• The ESX Server system can support 25% more OPMs when running

three VMs as compared to two VMs. The 25% increase in throughput is achieved by a 22% increase in CPU utilization.

• Hyper-Threading helps support more OPMs because the ESX Server

system can take advantage of four LCPUs (Logical CPUs) and thus allocate more CPU cycles to the Virtual CPU(s) assigned to the VMs.

• A larger L2 CPU cache can decrease the ESX Server host CPU

utilization by as much as 20%. This decrease can provide enough resources to host an additional virtual machine and thus provides for better utilization of the ESX Server host.

Abstract

Key Findings

October 2005 Page 3

1. Introduction …………………………………………………………………………… 4 2. Putting the components together 2.1 Connecting the hardware components …………………….5 2.2 Configuring the SAN ………………………………………………….7 2.3 Installing ESX server 2.5.1…………….…………………….……7 2.4 VM creation and installation of guest OS ….………….….7

2.5 Installation of MySQL and PHP ………………………………….8 2.6 Setting up the database …………………………………………….8

3. LAMP Performance and Sizing 3.1 Effect of Hyper-Threading………………………………………….9 3.2 How many VMs per ESX server? ………………….………………10 3.3 Effect of increasing L2 cache ………………………….…………11 4. Conclusion……………………………………………………………………………….14 5. Acknowledgements…………………………………………………………………14 Appendix A. Configuring LAMP stack ………….…………………………….15

Contents

October 2005 Page 4

LAMP (Linux, Apache, MySQL and PHP) is considered to be a proven web platform offering from the Open Source community. Recently, virtual machines (VMs) running unmodified guest operating systems on top of VMware ESX Server have gained attention and popularity. Running the Open Source LAMP stack on these VMs offers a solid and robust platform for small and medium sized enterprises to host web based application development, with acceptable price/performance ratios derived from the flexibility of virtual platforms, such as server consolidation and enhanced disaster recovery.

To further explore these factors, the Scalable Enterprise team at Dell conducted performance characterization and sizing tests of VMs hosted on ESX Server 2.5.1 running on Dell PowerEdge 1855 blade servers. The LAMP stack components, PE1855 server, and Dell/EMC storage were configured to support an OLTP workload environment. To generate the load, a Dell-developed tool – the Online DVD Store application1 - was used. The number of VMs running on the ESX Server host was gradually increased along with the OLTP workload per LAMP VM with the goal to achieve a high level of VM and ESX Sever utilization.

By running a simultaneous number of VMs each under an OLTP workload, Dell engineers were able to demonstrate how an ESX Server running on a Dell PowerEdge 1855 blade server could support increasing amounts of work while still effectively utilizing the underlying physical server and storage resources. This paper provides the findings from the performance and sizing tests and by using the analysis of the test results, Systems Engineers and IT Architects can leverage the information provided in this paper to help them to achieve effective workload throughput and virtual machine utilization.

1 Online DVD Store is a Dell-created online e-commerce test application. This application has a database component, a web application layer and a multi threaded driver program to drive the online transaction processing. The online DVD store can be downloaded from http://linux.dell.com/dvdstore/.

1. Introduction

October 2005 Page 5

This section describes how the Dell engineers designed, and deployed ESX Server 2.5.1 on the Dell PowerEdge 1855 blade server. The hardware components used are listed, the test environment architecture is provided, and specific steps used to deploy and configure VMware ESX Server 2.5.1, guest operating systems, and the LAMP stack components are explained. In addition, the steps taken to set up and create the database are explored. 2.1 Connecting the Hardware Components

Table 1 describes the details about the hardware components in the test environment. Table 1: Hardware components Component Configuration Server PowerEdge 1855 Blade Server Processor 3.6 GHz Intel® Xeon™, 2MB L2 Cache + Hyper-

Threading Memory 4 GB of system memory Storage Dell/EMC CX 400 with 15 x FC2 disks Switch Brocade Silkworm 3850 Clients PowerEdge 750 Network PowerConnect 5224

Figure 1 describes the way in which the hardware components were

connected.

2. Putting the Components Together

October 2005 Page 6

October 2005 Page 7

2.2 Configuring the SAN

After setting the appropriate fibre channel switch zoning and Navisphere® storage group configurations, four LUNs were created. Each LUN was configured as RAID 0 and was made of three 66GB 10K RPM disks. One of the LUNs was used to create a RAID 5 VMFS volume for the ESX Server system, while the other three LUNs were dedicated to the virtual machines using raw disk mapping (RDM)2. One of the key advantages of using raw devices is the ability of databases to deploy sophisticated data management algorithms with raw devices for maximum performance.

2.3 Installing ESX Server 2.5.1

ESX Server 2.5.1 software was installed on the PowerEdge 1855 blade server as per the recommended procedures described in Dell VMware ESX Server 2.5.1 Deployment Guide3. The details of the ESX Server components installed are described in Table 2 below. Table 2: ESX components Component Configuration ESX Server 2.5.1-13057 VirtualCenter 1.2-11392

ESX Server software was installed on the internal PE1855 blade server 73 GB SCSI hard disks. The two 73GB disks were configured as a mirror for fault tolerance. One of the onboard gigabit network interfaces was dedicated to the service console and the other one was dedicated to virtual machines.

2.4 VM creation and Installation of Guest OS

After the installation of ESX Server software, two VMs were created using VMware VirtualCenter. Red Hat® Enterprise Linux 3 Update 4 (RHEL3 U4) was installed on both of the VMs. Each VM was configured with one virtual CPU and 512 MB of RAM, one virtual disk (a “.vmdk” file residing on the VMFS volume), and one physical disk with direct access to a system LUN using RDM. VMTools was installed on all guest operating systems.

2 Starting with the ESX Server 2.5 release, raw device mapping allows a special file in a VMFS volume to act as a proxy for raw devices. The mapping file contains the metadata used to manage and redirect disk access to a physical device. This type of mapping merges VMFS manageability with raw device performance. For more information on RDM please see http://www.vmware.com/pdf/esx/esx25_rawdevicemapping.pdf. 3 Find the ESX Server best practices for Dell PowerEdge servers at http://www.dell.com/downloads/global/solutions/vmware_251_deployment_guide.pdf.

October 2005 Page 8

2.5 Installation of MySQL and PHP MySQL 5.0 was installed from the pre-built binaries distributed as RPMs obtained from the MySQL web site. These RPMs are optimized and are recommended by Dell and MySQL for production use. PHP was compiled from sources available at the PHP web site. Apache web server was used to provide the web application layer, which comes by default with the RHEL3 U4 distribution. The details of these installations are given in Appendix A. The version information of the software used in the test is described in Table 3. Table 3: Apache, MySQL, and PHP Details Component Configuration Apache 2.0.46-44 - Comes default with RHEL3 U4 distribution. MySQL 5.0.7 - Binaries distributed as RPM from MySQL network

http://dev.mysql.com/downloads PHP 5.0.4 - Tarball available from www.php.net

2.6 Setting Up the Database

MySQL data files were stored on the system LUN using RAID Level 0 (striping only, with no redundancy). The index and table files were stored on the local disk. Since the local disk is a mirrored pair (RAID Level 1), index and table files resided on a protected volume. For production level LAMP deployments, system designers should consider using SAN based RAID Levels 5 or 10 LUNs for storing data files in addition to conducting regularly scheduled backups.4 The log file writes are generally sequential. Hence the log files were stored on the local disk only. For the database sizes tested, the I/O capabilities of the local disks were sufficient to handle the transaction log I/O activity. The DVD store application uses both MyISAM and InnoDB tables. The database parameters were tuned as described in Appendix A. To test the MySQL database performance in an OLTP environment, a medium sized database (size = 1 GB) was built using the build scripts provided with the Online DVD Store utility. Then these data files were loaded into the database. A multi-threaded Microsoft® Windows®-based driver application supplied with the Online DVD Store utility was used to simulate an online transaction processing workload. This driver program was run on Dell PowerEdge 750 client machines running Windows Server 2003 Enterprise Edition. To simulate a real world scenario, the think time was set for two seconds and the percentage of new users was configured at 20%. The parameter used for measuring the performance and scalability of ESX Server software and the LAMP VMs was orders per minute (OPM). OPM can also be

4 For higher levels of database I/O activity, system designers may want to consider using RAID 10 LUNs.

October 2005 Page 9

thought of as Transactions per Minute. To scale the workload, the number of threads, was increased gradually from 10 to 20, and then 30.

In this section, various factors that can impact and benefit ESX Server installations and their hosted VMs are explored. In particular, technologies like Intel Hyper-Threading, and processor cache sizes are examined. Results from tests while utilizing these technologies are presented and analysis is provided that can help systems engineers and IT architects make informed decisions about the deployment of appropriate number of VMs for OLTP workload and virtual machine scaling.

Running the Tests: The DVD Store driver program was launched from the PE 750 client machines after the database load completed. The driver program writes the operations per minute to the console in plain text format. For ESX Server, the CPU utilization was measured using esxtop and vmkusage5 while VM utilization was captured with VirtualCenter. The tests were repeated for each of the four different scenarios described below. In each of these scenarios, the number of threads (or simulated users) in the driver program was increased by increments of 10 users starting with 10 users up to 30 users per VM. Effect of Hyper-Threading: PowerEdge 1855 blade servers support the latest Intel Xeon processors with Hyper-Threading Technology. To measure the effect of Hyper-Threading on a two-processor server, two sets of tests were conducted. The first one was without Hyper-Threading and the second one was with Hyper-Threading enabled.

In the first scenario, each of the VMs was scheduled to run on physical processors 0 and 1. The service console also used the physical processor 0. In the second scenario, with Hyper-Threading enabled, the VMs were scheduled to run on all four processors. In each of the following scenarios presented, the OPM shown are per VM. The data obtained from the test runs are as follows.

5 esxtop and vmkusage are monitoring tools available with ESX Server software. More on these tools can be found at http://www.vmware.com/vcommunity/technology/performance.

3.0 LAMP Performance and Sizing

October 2005 Page 10

Effect of Hyper-Threading

0100200300400500600700800900

1 2 3

10 x # users

Oper

atio

ns

per

min

ute

(O

PM)

OPM (No HT)OPM (HT)

Figure 2: Effect of Hyper-Threading As shown in Figure 2 there is an approximately 8% improvement in OPM per VM with Hyper-threading enabled on a two-way PowerEdge blade server. The reason Hyper-Threading helps support more OPMs is that ESX Server can take advantage of four LCPUs (Logical CPUs) and thus allocate more CPU cycles to the Virtual CPU(s) assigned the VMs. In addition, there is more processing power available to the ESX Server kernel and service console related operations. When Hyper-Threading is enabled, the VM scheduler will schedule the two VMs to run on all four LCPUs. In the case where only two CPUs were exposed to the ESX Server system, the system had to balance the requirements of the service console, ESX Server kernel, and VM operations. Since the Online DVD Store application is a CPU-intensive workload, one of the impacts on the ESX Server host with Hyper-Threading disabled is fewer completed OPM per VM. Number of VMs per ESX Server System: A third VM was created with a virtual hardware configuration just like the other LAMP VMs. All the VMs were configured without Virtual SMP. The data obtained are as follows.

ESX Resource utilization for more VMs

0

20

40

60

80

100

1 2 3 4

10 x # users

% E

SX

CP

U u

tili

zati

on

% ESX CPU Utilization (1 VM)% ESX CPU Utilization (2 VMs)% ESX CPU Utilization (3 VMs)

Deleted: when

October 2005 Page 11

Figure 5: ESX Resource utilization for increase in number of VMs

Performance for increased number of VMs

0100200300400500600700800900

1 2 3 4

10 x # users

Ord

ers

per

min

ute

(O

PM

) OPM (1 VM)OPM (2 VMs)OPM (3 VMs)

Figure 6: Orders per minute for an increase in number of VMs From Figure 5 it is clear that with three virtual machines the ESX Server host reaches a CPU utilization of around 90%. While an ESX Server system can sustain more than 90% CPU utilization, systems designers and IT administrators should keep in mind that spare CPU cycles need to be kept available to handle bursts of workload. By keeping enough CPU cycles in reserve, ESX Server systems can provide enough CPU resources to the VMs in situations where a VMotion event or temporary spike in OPMs in one or more VMs on the same host may occur. For some IT organizations, a maximum targeted ESX Server host CPU utilization threshold may be 75%, while for others the goal may be for 95%. These maximum CPU utilization levels can be determined in a number of ways, one of which is throughput. To illustrate how throughput can be used as a determining metric for establishing maximum desired ESX Server system CPU utilization, note in Figure 6 the decrease in the OPM per VM as the number of VMs and number of users per VM increase. There is a slight decrease in orders per minute when the number of virtual machines is increased from two to three. This decrease in OPMs per individual VM as the number of VMs increases is to be expected since the ESX Server host has to sustain more work with a finite pool of system resources. As shown in Figures 5 and 6, if this small decrease in orders per minute per VM is acceptable, a maximum of three virtual machines can be hosted on a single PowerEdge blade server for an OLTP workload similar to that generated by the Online DVD Store application. Another way to use throughput as a metric to set desired system utilization is to evaluate the overall OPMs per ESX Server host rather than at the individual VM level. As shown in Figure 6, at two active VMs with each VM sustaining 30

October 2005 Page 12

simulated users, the aggregate ESX Server host OPM throughput is 1,530 (765 per VM). At three active VMs, the overall OPM throughput rate increases to 2,040 OPM (680 per VM). By adding a third VM and incurring slightly less throughput per VM, the ESX Server system can be more effectively utilized and support 25% more OPMs as compared to only running with two VMs. And as indicated in Figure 5, the 25% increase in aggregate system throughput is achieved by a 22% increase in ESX Server system average CPU utilization. At two active VMs, the ESX Server host CPU utilization was 69%, and at three active VMs, the host CPU utilization was 88%. These two methods, throughput per VM and throughput per ESX Server host; are techniques that use throughput to help determine efficient ESX Server system utilization and to establish quantifiable metrics that can be used to determine appropriate CPU utilization thresholds. It should be kept in mind, however, that the data presented in this paper are average values. With any random workload, peaks and valleys will occur during the normal course of a business day, so allowing for spikes in resource utilization is one key criterion that systems administrators should consider when determining how many VMs to run on an ESX Server host. Effect of increasing L2 cache: To observe the effect of L2 CPU cache size, the 3.6 GHz 1 MB L2 CPU cache processors were replaced with two 3.6 GHz 2 MB L2 CPU cache processors. All the tests were re-run with this set of processors, which had double the amount of L2 cache compared to the previous configurations. Hyper-Threading was enabled on both processors and all three virtual machines were configured with one virtual CPU. Table 4: Processor L2 Cache Size Impact on ESX Server host CPU Utilization

# Users

% ESX CPU Utilization (3 VM with 1 MB

L2 cache)

% ESX CPU Utilization (3 VM with 2 MB

L2 cache) 10 34 30 20 60 52 30 88 72

Table 4 shows that as the number of users (threads) increases, the benefits of the larger CPU L2 cache appear as a reduction in ESX Server host CPU utilization (when compared to the configuration with a smaller L2 cache). Since the Online DVD Store workload is random in nature and creates reads and writes to the underlying database system disks, larger CPU L2 cache can hold more data, thus reducing the need to access data on slower disk subsystems. Because data can be accessed more quickly in processor cache, CPU cycles are freed up to handle other ESX Server kernel scheduling tasks.

October 2005 Page 13

Effect of L2 Cache on ESX Resources

0

20

40

60

80

100

1 2 3

10 x # users

% o

f E

SX

CP

U

Uti

liza

tio

n

% ESX CPU Utilization (3 VMwith 1 MB L2)% ESX CPU Utilization (3 VMwith 2 MB L2)

Room for extra VM

Figure 7: ESX CPU utilization with larger CPU cache size Examining Figure 7 reveals that when three VMs are hosted on a system powered by a CPU having 2 MB L2 cache, the CPU utilization on the ESX Server system was around 70% in contrast to 88% with 1 MB L2 cache. As a general rule, increasing L2 cache improves ESX Server software’s ability to host an additional virtual machine on the ESX server. To test this, an additional VM was added to the ESX Server and its workload was increased to 30 users as with the other three VMs. Tests from running four LAMP VMs with varied number of users reveal that with the larger L2 cache CPUs, the ESX Server system can sustain up to four VMs while still supporting acceptable OPM throughput rates. This is graphically illustrated via the data shown in Figure 8.

October 2005 Page 14

Average OPM/VM for 3 VM/host and 4 VM/host

0100200300400500600700800

1 2 3

10 x # Users

OPM

Average OPM for 3VM/hostAverage OPM for 4VM/host

Figure 8: Average OPM/VM for 3 VMs/Host and 4 VMs/Host

As introduced above, it is clear from the data shown in Figure 8 that even when four virtual machines are hosted on the ESX Server, each of the virtual machines can support sustainable OPM transaction rates. The OPM shown in Figure 8 are an average per VM, rather than the aggregate for the ESX Server host. It should be kept in mind when evaluating the results presented in this paper that the VMs are under moderate to heavy loads. VMs running under lighter utilization levels could result in a higher consolidation ratio of VMs to per ESX Server host. As discussed in the section on Hyper-Threading, it should be kept in mind that the optimal number of VMs, regardless of VM utilization levels, that can be run on an ESX Server host depends ultimately upon desired factors like response times, throughput rates, and the overall host utilization levels that meet an IT organizations Service Level Agreement (SLA). Thus, even though possible as shown in Figure 8, systems administrators running ESX Server hosts in production level environments may choose not to deploy a fourth VM so that enough resources remain available to support spikes or a Virtual Machine failover via VMotion.

October 2005 Page 15

The Dell PowerEdge 1855 blade server with ESX Server 2.5.1 software provides a platform that is well suited for hosting a minimum of two virtual machines running a medium sized (Database size =1 GB) OLTP type of workload on a Linux, Apache, MySQL, and PHP stack. In cases where Hyper-Threading is enabled, ESX Server hosts benefit from performance improvements of up to 8% as the number of simultaneous users accessing the databases increase. Increasing the physical processor L2 cache size is recommended for hosting more VMs on the same ESX Server host or to decrease ESX Server host CPU utilization. Doubling the L2 cache (from 1 MB to 2 MB) decreases the ESX Server CPU utilization by 18-20%, providing enough resources to host an additional virtual machine at a relatively low performance penalty to the other VMs. These series of tests and analysis show that as the number of supported users per VM and the number of VMs increased, acceptable levels of OPM throughput were provided by the ESX Server system. Throughput per VM and aggregate ESX Server host throughput were two methods presented that can be used to evaluate and set appropriate ESX Server system CPU utilization levels. Understanding and tracking ESX Server utilizations levels for resources like CPU and using the methods presented are one technique that IT organizations can implement to help maintain or exceed SLA targets. The authors extend their thanks to Dave Jaffe and Todd Muirhead for their support and willingness to lend expertise with the Online DVD Store application.

Conclusions

Acknowledgements

October 2005 Page 16

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. Dell, OpenManage, PowerEdge, and PowerVault are trademarks of Dell Inc. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others. ©Copyright 2005 Dell Inc. All rights reserved. Reproduction in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell. Dell cannot be responsible for errors in typography or photography.

October 2005 Page 17

Configuring LAMP stack

1. Setting up Raw Disk Mapping: After creating the virtual machines, through ESX MUI add a second hard disk to each virtual machine. When prompted for the type of virtual disk please select the System LUN/Disk. In the next screen choose the appropriate LUN and assign a meaningful metadata file name. Detailed information of RDM is outside the scope of this paper. Fore more information on RDM please refer to http://www.vmware.com/pdf/esx/esx25_rawdevicemapping.pdf. This newly added hard drive is visible as /dev/sdb when the guest OS boots up. After the guest OS installation is over, login as root and create a partition of size 20GB using partition utility like fdisk. This partition will be recognized as /dev/sdb1. Now issue the following command so that MySQL user has read and write access to the raw device. # chown mysql.mysql /dev/sdb1 2. Setting up MySQL: After installing RHEL3 U4 follow these step by step. Step 1: Stop any running MySQL service by issuing the following command as root. # /etc/init.d/mysql stop Step 2: Remove any previous installation of MySQL. # rpm –-nodeps –e `rpm –qa|grep –i MySQL` Step 3: Install the MySQL binaries.

# rpm –ivh MySQL-client-5.0.7-0.i386.rpm MySQL-devel-5.0.7-0.i386.rpm MySQL-server-5.0.7-0.i386.rpm

Step 4: Edit the /etc/my.cnf and tune these parameters.

N.B : If you don’t have /etc/my.cnf file previously, copy the file /usr/share/mysql/my-large.cnf to /etc/my.cnf.

Parameter Value key_buffer_size 16M table_cache 32M read_buffer_size 2M read_rnd_buffer_size 4M myisam_sort_buffer_size 16M

Appendix A

October 2005 Page 18

Step 5: Now is the time to initialize the raw device. Start the MySQL database. Remember that this may take quite some time as MySQL has to initialize the 20GB of raw space. The progress can be monitored from /var/lib/mysql/hostname.err file. # /etc/init.d/mysql start Step 6: Change the parameter innodb_data_file_path from /dev/sdb1:20Gnewraw to /dev/sdb1:20Graw. Notice that the key word new has been removed. Then restart the MySQL. # /etc/init.d/mysql restart 3. Configuring PHP: Obtain the php-5.0.4 tarball from php.net and follow the step by step. Step 1: Untar the php source as root. # tar –zxvf php-5.0.4.tar.gz Step 2: Build the php module from source as root. # cd php-5.0.4

# ./ configure --with-apxs2=/usr/sbin/apxs2 --with-mysqli=/usr/bin/mysql_config # make # make install

4. Configuring the web server: Step 1: Add the following lines in /etc/httpd/conf/httpd.conf

KeepAlive On KeepAliveTimeout 50 ServerName hostname:80 AddType application/x-httpd-php .php

Step 2: Restart the apache service. # service httpd restart

thread_cache 250 innodb_data_file_path /dev/sdb1:20Gnewraw

October 2005 Page 19

5. Setting up the database: Obtain the online DVD store from http://linux.dell.com/dvdstore. Untar the archive and follow the step by step procedure described in the README present in ds2 directory.