Wmware Storage Vmotion
Transcript of Wmware Storage Vmotion
BEST PRACTICES FOR SUN STORAGE 6000 SERIESAND 2500 SERIES ARRAYS
VMware STORAGE VMotion
White PaperMarch 2009
Sun Microsystems, Inc.
Table of Contents
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
VMware Storage VMotion Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
What is Moved by VMware Storage VMotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
How VMware Storage VMotion Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
VMware Storage VMotion Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
VMware Storage VMotion Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
VMware Storage VMotion Uses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Storage Tiering with Sun Storage Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Sun Storage Array Performance and Scalability Management . . . . . . . . . . . . . . . . 8
Storage Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Sun Storage 6000 and 2500 series architecture guidelines for
VMware Storage VMotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
IntroductionVMware Storage VMotion, allows the user to perform live online migration of a
running Virtual Machine’s disks, or disk files from ESX Server’s perspective, from
internal disk drives on a server, within and across storage arrays without the need
for downtime while maintaining transaction integrity. The key difference between
traditional live storage migration and Storage VMotion is that Storage VMotion can
perform live online migration of the entire operating system, including Swap or page
files and any virtual machine related configuration and log files, from one storage
device to another.
The purpose of this document is to provide guidelines and best practices for using
VMware Storage VMotion with Sun Storage 6000 series, including the new ST6580 and
ST6780, and the Sun Storage 2500 series storage arrays. Although all Fibre Channel or
iSCSI capable Sun Microsystems Storage arrays listed in the VMware Storage/SAN
Compatibility Guide are supported with VMware Storage VMotion. The guide is
updated on a regular basis with new storage arrays added so it is highly recommended
that the online version of the guide be referred to prior to providing any architectural
recommendations.
VMware Storage VMotion BasicsStorage VMotion was introduced in VMware Virtual Infrastructure 3 release with ESX
Server 3.5 and vCenter 2.5, initially supporting Fibre Channel storage. Release of ESX
Server 3.5 update 1 added support for iSCSI storage.
What is Moved by VMware Storage VMotion• Virtual machine’s home directory
• Configuration file (vmx)
• Logs
• Swap file (vswp)
• Any snapshots of the virtual machine
• Any other files related to the virtual machine
• Virtual machine disks or luns (vmdk)
• The operating system disk
• Data disk(s)
The disks can be moved independently as needed or required. The OS disk can be
moved without having to move data disks or luns and any and all disks can be placed
on any disk/lun configuration on the particular ESX server.
Sun Microsystems, Inc.1 VMware Storage VMotion
How VMware Storage VMotion WorksVMware Storage VMotion requires VMware vCenter 2.5 or later, VMware VMotion
license enabled on the host and the VMware provided Remote Command-Line
Interface application or Virtual Appliance which can be downloaded from
VMware.com. There are VMware vCenter plug-ins available online which allow for a
GUI-based Storage VMotion capability but none are officially support by VMware. For
more details, please refer to the VMware VI3 Basic System Administration Guide and
Remote Command-Line Interface Installation and Reference Guide.
VMware Storage VMotion consists of six steps outlined below:
Sun Microsystems, Inc.2 VMware Storage VMotion
Step 1. Copy virtual machine home directory to new location. This step uses NFCcopier to copy the files over to the new location
Step 2. Self-VMotion to new virtual machine home. A self-VMotion uses the sameprocess as a normal VMotion operation. This step enables the use of the new swapfile and reopen other copied files such as the virtual machine config file and othersuch files.
Sun Microsystems, Inc.3 VMware Storage VMotion
Step 3. Take a disk-only snapshot. This creates a child disk which will now be usedto maintain ALL changes being made to the Virtual Machine data.
Step 4. Copy virtual machine disk to destination (read-only).
Sun Microsystems, Inc.4 VMware Storage VMotion
Step 5. Consolidate child disk created in step 4 into copied disk.
Step 6. Delete original virtual machine home and disks and enable read/write onnew copied disk.
VMware Storage VMotion Requirements• VMware vCenter with VMware VMotion license installed
• VMware VMotion must be enabled and properly configured on the ESX server
(Refer to VMware Basic Administration Guide)
– The virtual machine must not be using any device preventing VMotion (e.g.
floppy disk, CD-ROM)
– A physical network link for VMotion is not required for Storage VMotion
• The ESX server must have sufficient resources to perform all of the steps
outlined above:
– At least twice the amount of memory and CPU resources to accommodate a
second instance of the virtual machine on which Storage VMotion is to be
performed (see Steps 1 and 2 above). For example, if the virtual machine uses
1 vCPU and 500 MB of memory, there needs to 1 vCPU and 500 MB readily
available beyond what is being used by all running virtual machines to be able
to perform Storage VMotion operation on this particular virtual machine. This
doubling of memory and CPU resources is temporary while Storage VMotion steps
above are taking place but required.
– At least twice the amount of storage needs to be available at the source storage
to allow for a snapshot to be taken prior to copying the virtual machine disk to
the destination storage (see Steps 3 to 5 above)
• Virtual machine’s disks must be snapshot eligible as a snapshot is taken in Step 3
to create a child disk prior to copying the disk to the new location. Without this
capability Storage VMotion will not work.
• When using RDMs, RDMs in Virtual Compatibility Mode are supported with Storage
VMotion as these are snapshot capable. Snapshots can not be taken for RDMs in
Physical Compatibility Mode (refer to VMware Fibre Channel SAN Configuration
Guide and VMware VM vs. Physical Server I/O Performance Comparison document
for more information on RDMs).
• The ESX server must have access to both the source and destination storage
or datastore
• A maximum of four concurrent Storage VMotion migrations can occur
simultaneously on the same datastore
• Only Fibre Channel and iSCSI storage arrays are supported
• Virtual machines requiring Storage VMotion capability must not have network
connection to any internal virtual switches
Sun Microsystems, Inc.5 VMware Storage VMotion
Sun Microsystems, Inc.6 VMware Storage VMotion
NOTE: Storage VMotion of an RDM does not move data from one physical raw lun to another.It only moves the mapping file over to the location where the virtual machine config file(vmx) is moved. To move data from Lun 1 physically, a different technology such as RVMcan be used. (See Figure 1).
• Virtual machines requiring Storage VMotion capability must not have any cluster
relationship with another virtual machine as is the case when using a clustering
technology such as Microsoft Cluster Server or Sun Cluster at the virtual
machine level
• Simultaneous Storage VMotion and VMotion on the same virtual machine is not
allowed. A virtual machine’s datastores can be moved to a different storage first
using Storage VMotion and the virtual machine can be moved to a different ESX
server second or vice-versa.
VMware Storage VMotion Best Practices• Although not required, it is best practice to use the VMware Remote CLI Virtual
Appliance to perform Storage VMotion tasks via command line. There are GUI-based
VMware vCenter plug-ins available but are not supported by VMware.com.
• A lot of pre-planning need to take place prior to any Storage VMotion operation
– Plan for Storage VMotion during initial design and architecture of the ESX servers,
the SAN and the storage arrays
• Free memory, CPU and storage space in source LUN need to be available
• Ideally, use different host bus adapter ports for the source and destination storage
on the ESX server for maximum performance
Figure 1. Data on RDM luns do not move as part of Storage VMotion.
Sun Microsystems, Inc.7 VMware Storage VMotion
• If moving virtual machine disks on datastores residing on the same storage array,
use different target ports on the array for the source and destination luns for
maximum performance
– Coordination between storage, server and VMware administrators is key as all
components need to work seemlessly for a successful Storage VMotion operation
– It is strongly recommended that Storage VMotion be performed during off peak
hours for the specific virtual machine in order to avoid any noticeable
performance impact
VMware Storage VMotion Uses
Storage Tiering with Sun Storage Storage ArraysThe most compelling case for Storage Storage VMotion is made when the need to
migrate from legacy to latest array technologies arises in a virtualized infrastructure
without a considerable impact to availability. Especially if there is no array-to-array
replication available, as may be the case with customers migrating from a non-Sun
storage array to the new Sun Storage 6780 or 6580 array for instance. There are many
alternative methods to migrate data from one storage to another, some of which are
listed below:
• Use of remote replication technologies within the arrays such as Remote Volume
Mirroring for Sun Storage 6000 series arrays or TrueCopy for Sun Storage 9000 series
arrays. However, this technology requires additional feature licenses to be enabled
on the arrays, which the customer may not need beyond the scope of migrating the
data from older to newer arrays. And, the remote replication technology works only
within the same series of arrays so replication between Sun Storage 6000 series and
Sun Storage 9000 series is not available without the use of additional software.
NOTE: Storage VMotion is not a technology replacement for Disaster Recovery implementa-tions requiring data services such as Remote Replication to remote sites. Remote Replicationof LUNs is still required for Storage Level data replication to remote sites and for maintainingdata redundancy at the same site.
• Snap Shot technologies within an array can be used to migrate data from one lun to
another due to any number of reasons such as a new raid layout, for example
moving from RAID-5 to RAID-6. These snapshot technologies such as ShadowImage
for Sun Storage 9000 series or SnapShot for Sun Storage 6000 or 2500 array series
also may require a license feature as is the case with ShadowImage for Sun Storage
9000 series and require further configuration tasks to setup a mirror, sync the luns,
break the mirror and then perform further administration tasks at the ESX server
level for the secondary lun to be mapped to the same ESX server.
– Storage VMotion is also not a technology replacement for In-Storage Replication
Data Services. Storage VMotion does not create a copy of the data, rather
migrates virtual machine and its associated disk files between a given set of LUNs.
Sun Microsystems, Inc.8 VMware Storage VMotion
• Stop all I/O to luns to be migrated, create a full backup and perform a restore to the
destination array
– Possible issues include unnecessary and prolonged downtime, errors during
backups and/or restores, possible overloading of the backup infrastructure which
may take resources away from regular day-to-day backup requirements or having
to create or add new backup resources such as tape drives, etc.
To avoid all of these and many other much more complicated, task-oriented methods
to perform a simple lun migration from one array to another or a full migration of all
ESX server luns from one array to anther or to a different set of luns with in the same
array, the use of Storage VMotion is the obvious choice as it maintains uptime for the
virtual machines in any situation without having go through complex tasks of setting
up remote replication, backup and recovery or an internal snapshot technology.
Virtual machine disks resident on an iSCSI array can be migrated to Fibre Channel or
vice-versa. For instance, a Test Virtual Machine moving into a Production status can
simply be migrated from a test Sun Storage 2510 to a new Fibre Channel Sun Storage
6780 Production array using Storage VMotion.
Sun Storage Array Performance and Scalability ManagementVMware Storage VMotion can be used to move virtual machine disks in order to
achieve optimal performance and scalability for established SAN environments as well.
Consider an ESX host where all of the virtual machines currently reside on the internal
disk storage. All of the virtual machines are exposed to down time due to the server
being the single point of failure. If the customer decides to scale their current
configuration to two ESX servers and shared SAN and/or iSCSI storage, the
configuration becomes much more scalable avoiding single points of failures at
the server layer. (See Figure 2).
Figure 2. VMs residing on internal drives of an ESX server are exposed to SPoF.Shared Storage offers enhanced redundancy and availability.
Sun Microsystems, Inc.9 VMware Storage VMotion
The configuration can be further enhanced by adding arrays which would then allow
for complete redundancy at all levels. (See Figure 3). Virtual machine disks can be
moved seamlessly from internal server disks to the single Sun Storage 6140 and then
can be further scaled and divided between the Sun Storage 6140 and the Sun Storage
6780 based on the degree of data and virtual machine criticality. Data requiring high
performance and redundancy can be moved independently to the higher performance
more scalable ST6780 and other data and virtual machine disks can stay on the
ST6140. All of this data migration can be performed using VMware Storage VMotion.
Storage MaintenanceAnother use of Storage VMotion could be storage maintenance which may have an
impact on production I/O. Storage space reclamation is a major factor and another
compelling reason for use of Storage VMotion.
Figure 3. Low $$/GB storage should be used for non-essential data and High $$/GB,high performing storage for critical data.
Sun Microsystems, Inc.10 VMware Storage VMotion
Sun Storage 6000 and 2500 Series Architecture Guidelinesfor VMware Storage VMotionWhen architecting a storage solution for VMware, it is important to consider all
aspects of the software including Storage VMotion, which can have a considerable
impact on storage I/O performance when active. (See Figure 4).
Given the illustration above, some important best practices to consider:
• It is strongly recommended to separate Storage VMotion traffic from Production I/O
traffic at the HBA level to the Storage array port to avoid any possible performance
bottle necks
– Another reason to design the solution as such is so data moves from Source Lun to
Destination Lun using a completely different I/O path which will allow Storage
VMotion to complete the process quicker than if the same path was to be used to
moved hence possible creating a bottleneck. The label “Storage VMotion” path
and “Production” path is generic in terms of the illustration above.
Figure 4. If possible, isolate Storage VMotion I/O traffic from Production I/O byusing different Initiators/Targets.
Sun Microsystems, Inc.11 VMware Storage VMotion
– Sun Storage Common Array Manager software can be used to change “Preferred”
Controller for each Lun or Volume on the array which allows the I/O load to be
spread across controllers. (See Figure 5).
• Avoid moving VMware datastores from a lun within the same volume on an array.
For example, in the illustration above, moving a datastore from Lun 2 to Lun 3 is
not recommended as it will put unnecessary I/O stress on that volume. The given
move also defeats the purpose of using Storage VMotion as a tool to disperse data
effectively and decrease performance bottlenecks unless there was a specific reason
to perform this task such as the need for more space at the lun level.
• For the Sun Storage 2510 array, it is strongly recommended to use dedicated host
network interface and array port to avoid severe I/O bottlenecks when moving
datastores from or to the array. Since the array ports run at 1 GigE speeds, moving
datastores from a Fibre Channel array such as the Sun Storage 2540 could result in
very slow Storage VMotion operation.
• Since Storage VMotion only currently supports Fibre Channel and iSCSI storage
arrays, Sun Storage 2530 array is not supported with Storage VMotion
• Follow all Sun Storage and VMware best practices and guidelines to configure the
ST6000 series and ST2500 series arrays for use with VMware ESX servers
Figure 5.
Sun Microsystems, Inc.12 VMware Storage VMotion
References• Sun Storage 6580 and 6780 Release Notes –
http://dlc.sun.com/pdf/820-5776-11/820-5776-11.pdf
• Sun Storage 6580 and 6780 Hardware Installation Guide –
http://dlc.sun.com/pdf/820-5773-10/820-5773-10.pdf
• Sun Storage 2500 Series Array Hardware Installation Guide –
http://dlc.sun.com/pdf/820-0015-12/820-0015-12.pdf
• Sun Storage 6140 Array Hardware Installation Guide –
http://dlc.sun.com/pdf/819-7497-11/819-7497-11.pdf
• Sun Storage Common Array Manager Software Installation Guide –
http://dlc.sun.com/pdf/820-5747-10/820-5747-10.pdf
• VMware VI3 and Virtual Infrastructure Basic System Administration Guide –
www.VMware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_admin_guide.pdf
• VMware Storage/SAN Compatibility Guide –
www.VMware.com/resources/compatibility/pdf/vi_san_guide.pdf
• VMware Remote Command-Line Interface Installation and Reference Guide –
www.VMware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_rcli.pdf
• VMware Fibre Channel SAN Configuration Guide –
www.VMware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_san_cfg.pdf
VMware Storage VMotion On the Web sun.com
© 2009 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, and the Sun logo are trademarks or registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the United States and othercountries. Information subject to change without notice. SunWIN #559704 Lit. #SYWP14884-0 04/09
Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com