Rearchitecting Storage for Server Virtualization
-
Upload
stephen-foskett -
Category
Technology
-
view
1.910 -
download
3
Transcript of Rearchitecting Storage for Server Virtualization
![Page 1: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/1.jpg)
Rearchitecting Storage for Server VirtualizationStephen FoskettOct 21, 2010
![Page 2: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/2.jpg)
This is Not a Rah-Rah Session
![Page 3: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/3.jpg)
Agenda•First 45 minutes
▫Impact of hypervisors on I/O▫VM storage approaches▫VM connectivity options
•Break•Second 45 minutes
▫Storage features for VM▫Questions and comments
![Page 4: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/4.jpg)
Introducing Virtualization
![Page 5: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/5.jpg)
Poll: Who Is Using VMware?
VMware
None
MicrosoftOther
Virtualization Users
Source: A dozen analyst SWAGs
![Page 6: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/6.jpg)
Server Virtualization “The I/O Blender”
•Shared storage is challenging to implement
•Storage arrays “guess” what’s coming next based on allocation (LUN) taking advantage of sequential performance
•Server virtualization throws I/O into a blender – All I/O is now random I/O!
![Page 7: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/7.jpg)
Server Virtualization requires SAN and NAS•Server virtualization has transformed the
data center and storage requirements▫VMware is the #1 driver of SAN adoption
today!▫60% of virtual server storage is on SAN or
NAS▫86% have implemented some server
virtualization•Server virtualization has enabled and
demanded centralization and sharing of storage on arrays like never before!
![Page 8: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/8.jpg)
Server Virtualization Recoil•Dramatically increased I/O•“Detrimental” to storage utilization•Patchwork of support, few standards
▫“VMware mode” on storage arrays▫Virtual HBA/N_Port ID Virtualization (NPIV)▫Everyone is qualifying everyone and
jockeying for position•Befuddled traditional backup, replication,
reporting
![Page 9: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/9.jpg)
Three Pillars of VM Performance
![Page 10: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/10.jpg)
Poll: Does Server Virtualization Improve Storage Utilization?
![Page 11: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/11.jpg)
Hypervisor Storage Approaches
![Page 12: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/12.jpg)
Hypervisor Storage Options:Shared Storage
• Shared storage - the common/ workstation approach▫ Stores VMDK image in VMFS datastores
▫ DAS or FC/iSCSI SAN
▫ Hyper-V VHD is similar
• Why?▫ Traditional, familiar, common (~90%)
▫ Prime features (Storage VMotion, etc)
▫ Multipathing, load balancing, failover*
• But…▫ Overhead of two storage stacks (5-8%)
▫ Harder to leverage storage features
▫ Often shares storage LUN and queue
▫ Difficult storage management
VMHost
GuestOS
DAS or SANStorage
VMFS VMDK
![Page 13: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/13.jpg)
Hypervisor Storage Options:Shared Storage on NFS
• Shared storage on NFS – skip VMFS and use NAS▫ NTFS is the datastore
• Wow!▫ Simple – no SAN
▫ Multiple queues
▫ Flexible (on-the-fly changes)
▫ Simple snap and replicate*
▫ Enables full Vmotion
▫ Use fixed LACP for trunking
• But…▫ Less familiar (3.0+)
▫ CPU load questions
▫ Default limited to 8 NFS datastores
▫ Will multi-VMDK snaps be consistent?
VMHost
GuestOS
NFSStorage
VMDK
![Page 14: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/14.jpg)
Hypervisor Storage Options:Raw Device Mapping (RDM)• Raw device mapping (RDM) -
guest VM’s access storage directly over iSCSI or FC▫ VM’s can even boot from raw devices
▫ Hyper-V pass-through LUN is similar
• Great!▫ Per-server queues for performance
▫ Easier measurement
▫ The only method for clustering
• But…▫ Tricky VMotion and DRS
▫ No storage VMotion
▫ More management overhead
▫ Limited to 256 LUNs per data center
VMHost
GuestOS
SAN Storage
Mapping File
I/O
![Page 15: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/15.jpg)
Physical vs. Virtual RDMVirtual Compatibility
Mode• Appears the same as a
VMDK on VMFS• Retains file locking for
clustering• Allows VM snapshots,
clones, VMotion• Retains same
characteristics if storage is moved
Physical Compatibility Mode
• Appears as a LUN on a “hard” host
• Allows V-to-P clustering,a VMware locking
• No VM snapshots, VCB, VMotion
• All characteristics and SCSI commands (except “Report LUN”) are passed through – required for some SAN management software
![Page 16: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/16.jpg)
Physical vs. Virtual RDM
![Page 17: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/17.jpg)
Which VMware Storage Method Performs Best?
Mixed Random I/O CPU Cost Per I/O
Source: “Performance Characterization of VMFS and RDM Using a SAN”, VMware Inc., 2008
VMFS,RDM (p), or RDM (v)
![Page 18: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/18.jpg)
Storage Connectivity Options
![Page 19: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/19.jpg)
Which Storage Protocol To Use?•Server admins don’t know/care about
storage protocols and will want whatever they are familiar with
•Storage admins have preconceived notions about the merits of various options:▫FC is fast, low-latency, low-CPU, expensive▫NFS is slow, high-latency, high-CPU, cheap▫iSCSI is medium, medium, medium, medium
![Page 20: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/20.jpg)
vSphere Protocol Performance
![Page 21: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/21.jpg)
vSphere CPU Utilization
![Page 22: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/22.jpg)
vSphere Latency
![Page 23: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/23.jpg)
Microsoft Hyper-V Performance
![Page 24: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/24.jpg)
The Upshot: It Doesn’t Matter• Use what you have and are familiar with!• FC, iSCSI, NFS all work well
▫Most enterprise production VM data is on FC, many smaller shops using iSCSI or NFS
▫Either/or? - 50% use a combination• For IP storage
▫Network hardware and config matters more than protocol (NFS, iSCSI, FC)
▫Use a separate network or VLAN▫Use a fast switch and consider jumbo frames
• For FC storage▫8 Gb FC/FCoE is awesome for VM’s▫Look into NPIV▫Look for VAAI
![Page 25: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/25.jpg)
Break Time!
Stephen [email protected]/sfoskett+1(508)451-9532
FoskettServices.comblog.fosketts.net
GestaltIT.com
25
![Page 26: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/26.jpg)
VMware Storage Features
![Page 27: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/27.jpg)
What’s New in vSphere 4•VMware vSphere 4 (AKA ESX/ESXi 4) is a
major upgrade for storage▫Lots of new features like thin provisioning,
PSA, Any-to-any Storage VMotion, PVSCSI▫Massive performance upgrade (400k IOPS!)
•vSphere 4.1 is equally huge for storage▫Boot from SAN▫vStorage APIs for Array Integration (VAAI)▫Storage I/O Control (SIOC) aka DRS for
Storage
![Page 28: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/28.jpg)
Storage Features By License
![Page 29: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/29.jpg)
Native VMware Thin Provisioning
• VMware ESX 4 allocates storage in 1 MB chunks as capacity is used▫Similar support enabled for virtual disks on NFS in
VI 3▫Thin provisioning existed for block, could be
enabled on the command line in VI 3▫Present in VMware desktop products
• vSphere 4 fully supports and integrates thin provisioning▫Every version/license includes thin provisioning▫Allows thick-to-thin conversion during Storage
Vmotion• In-array thin provisioning also supported (we’ll
get to that…)
![Page 30: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/30.jpg)
Pluggable Storage Architecture• VMware ESX includes multipathing
built in▫ Basic native multipathing (NMP) is
round-robin fail-over only – it will not load balance I/O across multiple paths or make more intelligent decisions about which paths to use
▫ E+ Only: vSphere 4 Pluggable Storage Architecture allows third-party developers to replace ESX’s storage I/O stack
• There are two classes of third-party plug-ins:▫ Path-selection plugins (PSPs)
optimize the choice of which path to use, ideal for active/passive type arrays
▫ Storage array type plugins (SATPs) allow load balancing across multiple paths in addition to path selection for active/active arrays
• EMC PowerPath/VE for vSphere does everything
![Page 31: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/31.jpg)
vStorage APIs for Array Integration (VAAI)• vSphere 4.1 only!• Array-based “Full Copy”
command offloads operations to array snapshots for Storage VMotion
• Acceleration of storage I/O - think "I/O" dedupe (not to be confused with data deduplication
• Hardware-assisted locking on a block-by-block basis (rather than entire LUN)
• Array-based thin provisioning integration using TRIM, zeroing, etc
• Supposed to have thin provisioning stun, but it’s AWOL
![Page 32: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/32.jpg)
Storage I/O Control (SIOC)• “SIOC provides a dynamic control
mechanism for proportional allocation of shared storage resources to VMs running on multiple hosts”
• ESX can provide quality of service for storage access to virtual machines▫ Enabled on the datastore object, when
a pre-defined latency level is exceeded on a VM it begins to throttle I/O based on the shares assigned to each VM
▫ SIOC is aware of the storage array device level queue slots as well as the latency of workloads and decides how it can best keep machines below the predefined latency tolerance by manipulating all the ESX host I/O queues
▫ Introduce an element of I/O fairness across a datastore
• But:▫ vSphere 4.1 and Enterprise Plus only▫ Only supported with block storage (FC
or ISCSI)▫ Does not support RDM’s or datastores
constructed of extents, only 1:1 LUN to datastore mapping
![Page 33: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/33.jpg)
Why NPIV Matters• N_Port ID Virtualization (NPIV)
gives each server a unique WWN▫ Easier to move and clone*
virtual servers ▫ Better handling of fabric login▫ Virtual servers can have their
own LUNs, QoS, and zoning▫ Just like a real server!
• When looking at NPIV, consider:▫ How many virtual WWNs does
it support? T11 spec says “up to 256”
▫ OS, virtualization software, HBA, FC switch, and array support and licensing
▫ Can’t upgrade some old hardware for NPIV, especially HBAs
Virtual Server
Virtual Server
Virtual Server
21:00:00:e0:8b:05:05:04
Without NPIV
Virtual Server
Virtual Server
Virtual Server
…05:05:05
With NPIV
…05:05:06 …05:05:07
![Page 34: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/34.jpg)
Intel VMDq, VMDc, MS RSS• VMDq is like NPIV for
network cards▫ Hardware-assisted sorting
of virtual network cards▫ Uses MAC address▫ Requires special driver
• Supported on ESX and Hyper-V
• Two more technologies:▫ VMDc is different – Intel’s
networking take on SR-IOV
▫ Microsoft RSS allocates work to multiple CPU cores
![Page 35: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/35.jpg)
And Then There’s VDI…•Desktop Virtualization (VDI) takes
everything we just worried about and amplifies it▫Massive I/O crunches▫Huge duplication of data▫More wasted capacity▫More user visibility▫More backup trouble
![Page 36: Rearchitecting Storage for Server Virtualization](https://reader031.fdocuments.in/reader031/viewer/2022020116/555c3b1dd8b42a2c068b4992/html5/thumbnails/36.jpg)
Thank You!
Stephen [email protected]/sfoskett+1(508)451-9532
FoskettServices.comblog.fosketts.net
GestaltIT.com
36