IBM XIV Gen3 Storage System

IBM XIV Gen3 Storage System Storage built for VMware vSphere infrastructures

Peter Kisich IBM Systems and Technology Group ISV Enablement

October 2011

© Copyright IBM Corporation, 2011

IBM XIV Gen3 Storage System

Table of contents Abstract........................................................................................................................................1 Introduction .................................................................................................................................1 New in the IBM XIV Gen3 Storage System................................................................................1 Key integration with vSphere.....................................................................................................2 VAAI..............................................................................................................................................3

Full copy................................................................................................................................................... 3 How effective is the offload process? ...................................................................................................... 4 Hardware-assisted locking (atomic test and set)..................................................................................... 4 Block zeroing (write same) ...................................................................................................................... 6

vStorage APIs for Array Awareness..........................................................................................6 IBM System Storage Plug-in for vCenter ..................................................................................7 Storage Replication Adapter (SRA)...........................................................................................8 Snapshot management in vSphere .........................................................................................10 QoS options...............................................................................................................................11 Summary....................................................................................................................................12 Appendix A: Ordering options for the XIV Gen3 Storage System........................................13 Appendix B: SAN ordering considerations ............................................................................16 Resources..................................................................................................................................18 About the author .......................................................................................................................18 Trademarks and special notices..............................................................................................19

1

Abstract This paper provides information about integration, feature description, and ordering considerations for the IBM XIV Gen3 Storage System for VMware vSphere 4 and 5 infrastructures.

Introduction

The IBM® XIV® Gen3 Storage System employs an advanced auto-balancing architecture built for today’s dynamic data centers with an eye towards tomorrow. With industry-leading storage software and a high-speed InfiniBand® grid, the XIV Gen3 system delivers storage features and required performance

demanded in VMware infrastructures including:

VMware vStorage APIs for Array Integration (VAAI) vStorage APIs for Storage Awareness (VASA) integration

Storage Replication Adapter for Site Recovery Manager Centralized management in vCenter Quality of service (QoS) features for mixed workload priorities

Automation and simplicity for virtual infrastructures Co-development for VMware vSphere 5, and beyond

The global partnership with IBM and VMware, coupled with the forward-thinking architecture of the XIV

Gen3, provides a solid foundation for virtual infrastructures today and in the future.

New in the IBM XIV Gen3 Storage System

All data modules in the XIV Gen3 Storage System are interconnected with a high speed, low latency InfiniBand fabric setting a new standard of performance and response time. The XIV Gen3 system then automates the storage configuration of the high performance architecture to ensure that every disk

allocated to VMware will have optimal performance.

Infiniband Switching


Figure 1: Data stores optimally distributed across all spindles, processors, and cache

2

The XIV Gen3 Storage System from IBM provides an advanced grid architecture that is built for dynamic

virtual infrastructures. All volumes allocated to VMware are balanced across all processors, spindles and cache in the XIV system. Automated data distribution by the XIV Gen3 system provides a perfectly balanced configuration that avoids hot spots and ensures maximum performance.

Also, new in the XIV Gen3, is the introduction of solid-state drive (SSD) slots that can enable an optional field upgrade of 7.5 TB of secondary cache in 2012. This secondary cache can provide an automated staging area for frequently accessed data improving performance for even the highest transactional and

database workloads.

Key integration with vSphere

Along with the great synergy provided by the enhanced XIV Gen3 architecture, IBM provides integrated software for vSphere for every XIV system such as:

VAAI

vStorage APIs for Storage Awareness IBM Storage Management Console for VMware vCenter Storage Replication Adapter (SRA) for Site Recovery Manager

Snapshot management in vCenter QoS for mixed workload priorities


3

VAAI

Virtualization administrators look for ways to improve scalability, performance, and efficiency of their vSphere infrastructure. One way is by utilizing storage integration with VMware VAAI. VAAI is a set of APIs or primitives that allow vSphere infrastructures to offload processing of data-related tasks, which can

burden a VMware ESX server. Utilizing a storage platform with VAAI enabled, similar to the IBM XIV Storage System, can provide significant improvements in vSphere performance, scalability, and availability.

Adding the VAAI driver for XIV enables the following primitives:

Full copy (also known as hardware copy offload): − Benefit: Considerable boost in system performance and fast completion of copy

operations; minimizes host processing and network traffic Hardware-assisted locking (also known as atomic test and set): Replacement of the

SCSI-2 lock/reservation in Virtual Machine File System (VMFS)

− Benefit: Significantly improves scalability and performance Block zeroing (also known as write same)

− Benefit: Reduces the amount of processor effort, and input/output operations per second (IOPS) required to write zeroes across an entire EagerZeroedThick (EZT) Virtual Machine Disk (VMDK)

XIV system now provides full support for VAAI. The following sections describe each of these primitives.

Full copy

Tasks such as Virtual Machine (VM) provisioning and VM migration are part of everyday activities of most VMware administrators. As the virtual environment continues to scale, it is important to monitor the overall impact that these activities have on the VMware infrastructure.

Toggle hardware assisted copy by changing the DataMover.HardwareAcceleratedMove parameter in the Advanced Settings tab in vSphere Virtual Center to 1.

When the value for hardware acceleration is 1, the data path changes for tasks such as Storage vMotion,

as illustrated in Figure 2.


4

Figure 2: Full copy primitive

Here, the ESX server is removed from the data path of the data copy when hardware copy is enabled. Removing copy transactions from the server workload greatly increases the speed of these copy functions while reducing the impact to the ESX server.

How effective is the offload process?

During IBM lab testing, data retrieved from the VMware monitoring tool, esxtop showed that commands per second on the ESX host were reduced by a factor of 10. Copy time reduction varies depending on the VM but is usually significant (over 50% for most profiles).

A few examples of this performance boost at customer data centers are shown in Table 1.

Customer Test Before VAAI After VAAI Time reduction (in percentage)

Major Financial 2 VMs 433 Sec 180 sec 59%

Electric Company 2 VMs 944 sec 517 sec 45%

Petroleum Company

40 VMs 1 hour 20 min 67%

Table 1: Field results for full copy

Full copy effect: Thousands of commands and IOPs on the ESX server are freed up for other tasks and promote greater scalability.

Hardware-assisted locking (atomic test and set)

Just as important as the demonstrated effect of hardware-assisted copy, the hardware-assisted locking

primitive also greatly enhances VMware cluster scalability and disk operations by preventing disk corruptions of the clustered file system (VMFS) with tighter granularity and efficiency.

It is important to understand why locking occurs in the first place. For block storage environments,

VMware data stores are formatted with VMFS. VMFS is a clustered file system that uses Small Computer


5

System Interface (SCSI) reservations to handle distributive lock management. When there is a change to the metadata of the file system by an ESX server, the SCSI reservation process ensures that shared

resources do not overlap with other connected ESX hosts by obtaining exclusive access to the logical unit number (LUN).

A SCSI reservation is created on VMFS when (not a complete list):

VMDK is first created VMDK is deleted VMDK is migrated

VMDK is created via a template A template is created from a VMDK Creating or deleting VM snapshots

VM is switched on or off

Although normal I/O operations do not require this mechanism, these boundary conditions have become more common as features such as vMotion with Distributed Resource Scheduler (DRS) are used more

frequently. This SCSI reservation design leads to early storage area network (SAN) best practices for vSphere to dictate a limit in cluster size for block storage (about 8 to 10 ESX hosts).

With hardware-assisted locking, the LUN locking processing is transferred to the storage system. This

reduces the number of commands required to access a lock, provides locks to be more granular, and leads to better scalability of the virtual infrastructure.

VMDK

VMDK

VMDK

VMDK

VMDK

VMDK

Figure 3: Atomic test and set primitive

Hardware-assisted locking effect: Hardware-assisted locking will increase VMs per data store, ESX servers per data store, and overall performance. This functionality coupled with 60 cores and 360 GB of memory for the XIV Gen3 Storage System yields best in industry scalability and performance.


6

Block zeroing (write same)

Block zeroing is designed to reduce the amount of processor and storage I/O utilization required to write

zeroes across an entire EZT VMDK when it is created. With the block zeroing primitive, zeroing operation for EZT VMDK files are offloaded to the XIV Storage System without the host having to issue several commands.

Figure 4: the write same primitive

Block zeroing effect: Block zeroing reduced overhead and provided better performance for creating EZT

virtual disks. With XIV, EZT volumes are available immediately through fast write caching and de-staging.

VAAI support on XIV storage systems liberates value compute resources in the virtual infrastructure Offloading processor and disk intensive activities from the ESX server to the storage system provides

significant improvements in vSphere performance, scalability and availability.

Note: Before installing the VAAI driver for the XIV storage system, ensure 10.2.4a or higher is the installed microcode. For vSphere 5.x and higher, the VAAI driver is no longer required for IBM Storage.

vStorage APIs for Array Awareness

The IBM Storage provider for VMware VASA provides even more real-time information about the XIV

Storage System. The IBM XIV Storage System VASA provider for VMware vCenter adds:

Real-time disk status Real-time alerts and events from the XIV Storage System to vCenter

Support for multiple vCenter consoles and multiple XIV Storage Systems Continuous monitoring through storage monitoring service for vSphere Support for policy-driven storage profiles


7

Figure 5: VASA block diagram

Adding VASA support that is available in vSphere 5, allows VMware and Cloud administrators insights to

help provide improved availability, performance, and management of the storage infrastructure.

In addition to VASA, the XIV Storage System also provides a vCenter Plug-in for vSphere 4 and vSphere 5, which extends management of the storage to provisioning, mapping, and monitoring of

replication, snapshots, and capacity.

IBM System Storage Plug-in for vCenter

The IBM System Storage Plug-in for vCenter couples the XIV architecture advantages with powerful storage management tools. Features include:

Volume creation

Mapping and automated discovery Configuration optimization Capacity reports and monitoring

Snapshot details Replication status (RPO)

With these plug-in tools, management processes and operations can be streamlined in vSphere infrastructures.


8

The XIV GUI provides simplicity and functionality unmatched in the industry today. This leading and forward design has carried over to the vCenter plug-in. Combining the plug-in with the automation applied

to every LUN created on the XIV storage system ensures that storage provisioned from vCenter will always be optimally placed in the storage configuration.

Along with strong functionality, the IBM XIV vCenter Storage Plug-in maintains strong secure use of

storage administration tasks with multiple levels of access.

Storage Replication Adapter (SRA)

Site Recovery Manager is a VMware product for managing, testing and automating disaster recovery workflows.

The SRA is a critical part of Site Recovery Manager deployment, but it is supplied by the storage vendor

rather than VMware.

The XIV SRA not only provides the underlying disk-based replication with VMware awareness but also greatly simplifies disk replication without additional complexity and hardware components required by

other vendor solutions.


9

Through advanced XIV virtualization, storage replication configurations can be completed with drag and drop and right-click automation.

Figure 6: Drag-and-drop replication configuration

Site Recovery Manager provides powerful testing capabilities. Disaster recovery scripts can be tested

using XIV high performance and incremental snapshots at the remote site. These tests not only provide a deep understanding of the potential point in time recovery, which is recovery point objective (RPO), but also the time it takes to recover, which is recovery time objective (RTO).

The software for Site Recovery Manager integration is included with the XIV Storage System, and you can download it directly from the IBM website at: ibm.com/systems/storage/disk/xiv/


10

Snapshot management in vSphere

IBM Tivoli® Storage FlashCopy® Manager allows administrators to manage, schedule, and recover snapshots on the IBM XIV Storage Systems directly from the vCenter GUI.

Figure 7: vCenter console

Along with management capabilities, snapshots taken in VMware integrate with the VMware snapshot

process to ensure that all VMs are recoverable when restored.

When taking a snapshot in vCenter:

FlashCopy Manager initiates a VMware software snapshot of the VM through the vSphere

API FlashCopy Manager determines the LUN(s) that are associated with the virtual machine FlashCopy Manager invokes a hardware snapshot, creating a persistent copy of the virtual

disk(s) and associated VMware snapshots The hardware snapshot is kept for restore until specified for deletion through user-created

policies

Figure 8: Snapshot process

IBM XIV Storage System performs snapshots with virtually no overhead with re-direct on write technology. Snapshots are made without the overhead associated with copy on write snapshot processing used by other vendor solutions. Adding to this efficiency, XIV incremental snapshots are optimally configured (just


11

as primary volumes) ensuring enterprise-class read and write performance while eliminating the need for processing and space overhead.

QoS options

As data center consolidation rates accelerate, there is a larger demand to apply different priorities to

VMware workloads and better balance workloads on storage devices. These features include:

Storage I/O Control (SIOC) Storage DRS

In addition to full support for these features, the XIV Storage System provides an always autobalanced storage configuration. If throttling particular storage is required, the XIV also provides advanced and tunable QoS features on the storage system.

The combination of VMware features and storage features for QoS provides the maximum flexibility and granularity when applying restricted or limited storage resource allocation to vSphere.

SIOC

With SIOC, VMware administrators can apply restricted storage performance access to a virtual disk on a

data store in vSphere. This function fits well with the always balanced disk performance provisioned for the XIV Gen3 Storage System. That is, with the XIV, storage is physically optimized by default and vSphere administrators need to restrict only virtual disks and machines that might conflict with higher priority virtual

systems.

Storage DRS

Storage DRS establishes data store clusters. The data store clusters allow the dynamic transfer of virtual disks to different data stores according to the policies set in DRS. The XIV Storage System supports storage DRS but also provides a more global approach to load balancing.

All storage created for vSphere on the XIV system is balanced across all processors, ports, spindles, and cache modules in the XIV Storage System. The XIV Gen3 storage configuration in optimal balance prevents hitting thresholds and triggers that set off redistribution of disk resources.

Storage QoS on the XIV Storage System

By default, the XIV Storage System provides optimal performance for all the disks provisioned to vSphere.

There might be requirements for limiting host performance to the disk. With the XIV system, you can restrict the IOPS or bandwidth provided to a host or group of hosts.

The XIV system gives you the choice of utilizing VMware QoS features or off-loading performance

management to the hardware. Scalable QoS in XIV Gen3 gives cloud administrators maximum flexibility for managing storage resources across the diverse workloads in consolidated vSphere infrastructures.


12

Summary

The IBM XIV Gen3 Storage System provides a solid foundation for vSphere. The automation provided by the XIV system reduces the steps required to realize outstanding storage performance and functionality for VMware environments. This coupled with a strong global partnership between VMware and IBM ensures

that the IBM XIV Gen3 Storage System continues to lead for vSphere infrastructures today and in the future.

You can find more information about the IBM XIV Gen3 at: ibm.com/systems/storage/disk/xiv/


13

Appendix A: Ordering options for the XIV Gen3 Storage System

The XIV Gen3 Storage System is ordered according to the capacity required. The minimum physical

installation is six data modules yielding 55 TB of usable storage (there is also a Capacity on Demand (CoD) plan allowing a three-module 28 TB minimum entry point). Although 28 TB is the minimum configuration for XIV using CoD, it is appropriate only for storage requirements that will grow to at least

55TB in a one- to three-year period.

Figure 9: XIV Gen3 Internal Components


14

Initial configuration options for the XIV Gen3 are shown in Table 2.

Number of

data modules 6 9 10 11 12 13 14 15

Number of disks 72 108 120 132 144 156 168 180

Usable capacity TBs (2-TB drives)

55 88 102 111 125 134 149 161

Usable capacity TBs (3-TB drives)

84 132 154 168 190 203 225 243

Fibre Channel (FC) ports (8 GB)

8 16 16 20 20 24 24 24

Processor (Quad core)

6 9 10 11 12 13 14 15

Memory (GB) 144 216 240 264 288 312 336 360

Table 2: Configuration options for XIV Gen3 Storage System

If ordered initially with less than 15 data modules, the XIV Gen3 Storage System can be upgraded using the increments in the above table. Additional capacity, processing, connectivity and cache can all be

added on-the-fly without disruption or performance impact.

Capacity on demand (CoD)

With the CoD ordering option for the XIV Gen3 Storage System, customers benefit from the full processing, caching, and drive spindle performance of the physically deployed system; yet only an initial usable capacity is purchased.

The minimum usable capacity purchased in a CoD configuration is 28 TB (other initial increments are available). This minimum entry point consists of three CoD data modules and three CoD I/O data modules. As shown in Figure 10, this configuration provides 28 TB of purchased capacity and 27 TB of in-place

expansion.

Although there is a slight premium per usable TB for CoD options, the performance of CoD purchased capacity benefits from the additional processor, memory, FC ports, and spindles in a base 55 TB

configuration.


15

0

10

20

30

40

50

60

TBu

Un-act.

Active

Figure 10: Available in-place upgrade with CoD

For more information about the CoD program as well as terms and conditions, contact your IBM representative or authorized partner.


16

Appendix B: SAN ordering considerations

The SAN ordering considerations include queue depth, multipathing, and zoning and cabling.

Queue Depth

In general, the queue depth setting in the host bus adapter (HBA) has to be extended as consolidation scales higher and I/O requirements are high. Refer to IBM Redbooks® mentioned in the “Resources”

section.

Multipathing

Round robin policy should be implemented for each LUN from the XIV. This will achieve the best

performance results.

VMware 4.1 update 1 and higher versions will automatically detect XIV and set the round robin policy.

VMware 5.x and higher versions can automatically detect XIV and set the round robin policy.

Zoning and cabling

As the XIV Storage System is an active grid, ensure that the workload from VMware is balanced across all the six I/O controllers. A typical cabling diagram is shown in Figure 11.

All the paths are

active and use native VMware round robin multipathing.

Figure 11: A typical cabling diagram


17

In this example, a zone should be created for each initiator. Depending on the SAN vendor recommendations, all the six paths can then be added to that zone or separate zones can be created for

each initiator / target path. Consult your SAN fabric vendor for best practices.

The end results provide a path from each host initiator to all the six targets cabled in the diagram.


18

Resources

The following websites provide useful references to supplement the information contained in this paper:

IBM Systems on PartnerWorld ibm.com/partnerworld/systems

IBM Redbooks

ibm.com/redbooks

XIV Storage System: Host Attachment and Interoperability

ibm.com/redbooks/abstracts/sg247904.html

IBM Publications Center

www.elink.ibmlink.ibm.com/public/applications/publications/cgibin/pbi.cgi?CTY=US

About the author

At IBM, Peter Kisich works closely with storage development organizations and field solutions architects to ensure that IBM technologies can continue to innovate and build solutions for enterprise cloud

environments. An accomplished speaker, Mr. Kisich presents the value of IBM storage for virtual infrastructures to the global community of IBM customers, Business Partners and employees.

Before IBM, Pete Kisich served as a Technical Alliance Manager for VMware, supporting storage partners

NetApp, Hitachi, and Oracle.


19

Trademarks and special notices

© Copyright IBM Corporation 2011. All rights Reserved.

References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these

symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at

www.ibm.com/legal/copytrade.shtml.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States,

other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

SET and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC.

Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly

available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the

supplier of those products.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the

full text of the specific Statement of Direction.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to

any future products. Such commitments are only made in IBM product announcements. The information is


http://www.ibm.com/legal/copytrade.shtml


20

presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the

storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here.

Photographs shown are of engineering prototypes. Changes may be incorporated in production models.

Any references in this information to non-IBM websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk.

IBM XIV Gen3 Storage System

Technology

Transcript of IBM XIV Gen3 Storage System