Virtual VolumesTechnical Deep Dive

Ken Werneburg, VMware, IncPatrick Dirks, VMware, Inc

STO4649

#STO4649

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

CONFIDENTIAL 2

Today’s Challenges in External Storage ArchitecturesExternal Storage

vSphere

Array-a Array-b

Today1. Create fixed-size, uniform LUNs

2. Lack of granular control

3. Complex Provisioning cycles

4. LUN centric storage configurations

✖  Extensive manual bookkeeping to match VMs to LUNs✖ LUN-granularity hinders per-VM SLAs✖ Overprovisioning (better safe than sorry!)✖ Wasted resources, wasted time, high costs✖ Frequent Data Migrations

LUN

LUNLUN

LUNLUN

CONFIDENTIAL 3

5

Instead… an App-centric Model Drives Agility and QoS

App-centric Automation

• Dynamic delivery of storage service levels when needed

• Fine control of data services at the VM level

• Common management across heterogeneous devices

Rapid provisioning No overprovisioning

of resources

QoS automation Simple change

management

Today

• Static pre-allocation of shared storage container (LUN)

• Data services tied to storage container

• Vendor specific management

✖ Long provisioning cycles✖ Overprovisioning of

resources

Today’s Infrastructure-centric Model

✖ Management complexity ✖ Frequent data migrations

CONFIDENTIAL

Replication Snapshots Caching Encryption De-duplication

vSphere

External Storage Architectures with Virtual VolumesExternal Storage with VVols

Policy based Management

Offloaded Data Services

Eliminates LUN Management

Provides Per-VM Granularity

datastore a

External Storage without VVols

vSphere

Array-a Array-b

LUN

LUNLUN

LUNLUN

AND

CONFIDENTIAL 7

CONFIDENTIAL 9

Agenda

1 What are VVols?

2 Management Plane

3 Data Plane

4 Consumption Model - SPBM

5 VM Lifecycle with VVols

6 VVol Snapshots

7 Questions and Answers

vSphere Virtual VolumesA More Efficient Operational Model For External Storage

13

VMware vSphere Virtual VolumesIntegration Framework for VM-Aware Storage

VirtualVolumes

Overview

• Virtual disks are natively represented on arrays

• Enables VM granular storage operations using array-based data services

• Extends vSphere Storage Policy-Based Management to the storage ecosystem

• Supports existing storage I/O protocols (FC, iSCSI, NFS)

• Based on T10 industry standards

• Industry-wide initiative supported by major storage vendors

• Included with vSphere

VM

CONFIDENTIAL

High Level Architecture

Published Capabilities

Snapshot

Replication

Deduplication

Encryption

Overview

• No FileSystem

• ESX manages Array through VASA (vSphere APIs for Storage Awareness) APIs.

• Arrays are logically partitioned into containers, called Storage Containers

• VM disks, called Virtual Volumes, stored natively on the Storage Containers.

• IO from ESX to array is addressed through an access point called, Protocol Endpoint (PE)

• Data Services are offloaded to the array

• Managed through storage policy-based management framework

vSphereStorage Policy-Based Mgmt.

Virtual Volumes

Storage PolicyCapacity

Availability

Performance

Data Protection

Security

PE VASA Provider

CONFIDENTIAL 14

vSphere Virtual Volumes• Virtual Volumes

– Virtual machine objects stored natively on the array

– No Filesystem on-disk formatting required

• There are five different types of recognized Virtual Volumes:– CONFIG – vmx, logs, nvram, log files, etc

– DATA – VMDKs

– MEM – Snapshots

– SWAP – Swap files

– Other – vSphere solution specific type

vSphere Web Client View

vvol

CONFIDENTIAL 16

Storage Container Storage Containers

• Logical storage constructs for grouping of virtual volumes.

• Typically defined and setup by storage administrators on the array in order to define:

– Storage capacity allocations and restrictions

• Capacity is based on physical storage capacity

• Logically partition or isolate VMs with diverse storage needs and requirement

– Storage policy settings based on data service capabilities

• Minimum one storage container per array

• Maximum depends on the array

vSphere Virtual Volumes

SAN / NAS

Storage Containers

CONFIDENTIAL 17

Differences between Storage Containers and LUNs

• Size based on array capacity

• Max number of SCs depend only on the array ability

• Size of SC can be extended

• Can distinguish heterogeneous capabilities for different VMs (Virtual Volumes) provisioned in that SC

• Fixed size mandates more number of LUNs

• Needs a FileSystem

• Can only apply homogeneous capability on all VMs (VMDKs) provisioned in that LUN.

• Managed by In-band FileSystem commands

Storage Containers

LUN

CONFIDENTIAL 19

Storage Container (SC)vSphere Web

Client

Storage Management UI

Datastore

Storage Container

• What do the Admins see?• How are the storage containers setup?• What does the vSphere Admins see?• Why are we still creating datastores in this

new model?

CONFIDENTIAL 20

Virtual Volumes

VM objects view from a storage container on an arrayVM objects view from a datastore

vSphere Web Client Storage Management UIvSphere Admin View Storage Admin View

CONFIDENTIAL 22

Management PlanevSphere Virtual Volumes

VASA Provider (VP)• Software component developed by

storage array vendors

• ESX and vCenter Server connect to VASA Provider (VP) using standard VASA protocol

• VP provides Storage Awareness services and presents array’s SPBM capabilities

• Single VP can manage multiple arrays

• Interface between VP and array is private and VP can be designed to run anywhere (e.g. within the array’s management server, in firmware, or outside the array)

• Responsible for creating and managing Virtual Volumes

SAN / NAS

Virtual Datastore

StandardSAN/NASData Path

Protocol Endpoint(s)

vSphereVirtual Volumes

Storage Container

VASA Provider (VP)

Standard VASAControl Path

Private arrayControl Path

CONFIDENTIAL 24

PEPE

PE

Data PlanevSphere Virtual Volumes

Discovery Procedures – Storage Container

Virtual Volumes

Snapshot Replication

vCenter

VASA Provider

Storage Container Discovery Process• Storage admin sets up Storage Container

with desired capacity

• Desired Capabilities are configured for the Storage Containers

• VASA Provider (VP)’s URL is registered in VC

• VASA Provider presents Storage Container and its capabilities to vCenter

• Any new VMs that are created will subsequently be provisioned in the Storage Container based on SPBM profile associated with virtual disks

CONFIDENTIAL 26

Protocol Endpoints Protocol Endpoints• Access points that enables communication

between ESXi hosts and storage array systems.– Part of the physical storage fabric– Created by Storage administrators

Scope of Protocol Endpoints• Compatible with standard SAN and NAS

Protocols:- iSCSI- FC- FCoE- NFS v3

• Existing multi-path policies and NFS topology requirements are applied to the PE

Why Protocol Endpoints?

SAN / NAS

Virtual Datastore

DataPathProtocol Endpoint(s) PE

vSphereVirtual Volumes

Storage Container

CONFIDENTIAL 27

Discovery Procedures – Protocol EndpointProtocol Endpoint discovery process• VASA Provider (VP) reports PEs for container

to VC at registration

• When Virtual Datastore is created on host, VC passes PE information to host

• SCSI PEs are discovered during an ESX rescan

• NFS PEs are configured and mounted automatically

• Hosts report to VP which PEs are accessible

SAN / NAS

Virtual Datastore

DataPathProtocol Endpoint(s) PE

vSphereVirtual Volumes

Storage Container

VASA Provider

CONFIDENTIAL 29

Binding Operations• Bindings are data paths created by VP, in

coordination with the array, on request from ESXi hosts, between hosts and array for accessing virtual volume.

• Different Binding Mechanism:• Binding – requests creation of I/O channels

for a virtual volume

• Unbind – destroys the I/O channel for a virtual volume to a given ESXi host

• Rebind – provides the ability to change the I/O channel (PE) for a given virtual volumes run time using events.

create bind

unbinddelete

VM Creation VM power-onopen (2)

I/O read (2)

I/O write (2)

VM power-offclose (2)

VM destroy

VP rebalance

REBINDI/O

Virtual Volume Lifecycle

CONFIDENTIAL 30

Bind Requests

• IO path is established through a VASA Bind request

• VASA Provider does two things upon receiving a Bind request

• Coordinates with array to select a PE and unique secondary ID for VVol to be bound

• Returns a PE ID and secondary ID to which the VVol is bound to host

• For SCSI, the secondary ID is the secondary LUN ID

• For NAS, the secondary ID is the file path

• M:M Relationship between VVol and PE

Virtual Volumes

VASA Provider

VVol

PE

VVol

Bind

CONFIDENTIAL 32

CONFIDENTIAL

Virtual Volumes – Other Bind Scenarios

UNBIND

Virtual Volumes

VASA Provider

VVol

PE

VVol

UNBIND

• Delete binding of a VVol to the PE

• VASA Provider deletes binding on request of host (usualy at VM power-off)

REBIND

Virtual Volumes

VASA Provider

VVol

PE

VVol

PE

REBIND

• Move VVols access to different PE

• VASA Provider may choose to issue a rebind for several reasons, such as load balancing or storage migration

33

VVol Consumption Model:Ensuring SLOs through SPBM – Storage Policy-Based Management

Storage Capabilities and VM Storage Policies• Storage Capabilities – are array based features and

data services specifications that capture storage requirements that can be satisfied by a storage arrays advertised as capabilities

• Storage capabilities define what an array can offer to storage containers as opposed to what the VM requires

• Arrays Storage Capabilities are advertises to vSphere through the Vendor Provider and VASA APIs

• In vSphere Storage Capabilities are consumed via VM Storage Policy constructs

• VM Storage Policies is a component of the vSphere Storage Policy-based management framework (SPBM)

SPBM

object manager

virtual disk

Datastore ProfileVM Storage Policy

vSphere VM Storage Policy Management Framework

Storage Capabilities for Storage Array

Access

Capacity

Published CapabilitiesSnapshot

Replication

Deduplication

QoS

Virtual Datastore

CONFIDENTIAL 35

Storage Policy Based Management (SPBM) – Array Capabilities

Virtual Volumes APIs

Storage Policy-Based Mgmt.

CV

CV

CV

Storage admin

Publish Capabilities • Array based features and

data services• Defines what an Array can

offer • Advertised to ESX through

VASA APIs

Sample Default Profile for (6090a058-cd89-ffe3-87763007db37] - capabilities

Disk Types

Disk Encryption

Dedupe

Replication

Snapshot

CONFIDENTIAL 36

Storage Policy Based Management (SPBM) – VM Policies

CONFIDENTIAL 37

Storage Policy Based Management (SPBM)

CONFIDENTIAL 38

Management Workflow

• What do the Admins see?• How are the storage

containers setup?• What does the vSphere

Admins see?• Why are we still creating

datastores in this new model?

Storage policies

vSphere Web Client

Storage Management UI

Datastore

Storage Container

Storage Capabilities

virtual volumes

virtual machines

CONFIDENTIAL 39

Provision and Lifecycle Workflows

vvol

DATA

vvol

CONF

vvol

SWAP

vvol

DATA

vvol

CONF

vvol

SWAP

Provisioning Workflow

storage arrays

PE

vSphere Admin1. Create Virtual Machines2. Assign a VM Storage Policy3. Choose a suitable Datastore

Under the Covers• Provisioning operations are

translated into VASA API calls in order to create the individual virtual volumes.

Under the Covers• Provisioning operations are

offloaded to the array for the creation of virtual volumes on the storage container that match the capabilities defined in the VM Storage Policies

offload to arrayVirtual Datastore

Storage Container

vSphereVirtual Volumes

CONFIDENTIAL 41

Snapshots• Snapshots are a point in time copy on write

image of a Virtual Volume with a different ID from the original

• Virtual Volumes snapshots are useful in the contexts of creating:– a quiesced copy for backup or archival

purposes, creating a test and rollback environment for applications, instantly provisioning application images, and so on

• Two type of snapshots supported:– Managed Snapshot – Managed by ESX

• A maximum of 32 vSphere managed snapshot are supported for linked clones of an individual VM

– Unmanaged Snapshot – Manage by the storage array• Maximum snapshot dictated by the storage array

Managed Snapshot - vSphere

Unmanaged Snapshot - Array

CONFIDENTIAL 42

Snapshots: Files vs. Vvols

flatfilemain.vmdk main.vmdk

redologsnap.vmdk

snap.vmdkVVolID86

VVolID42

CONFIDENTIAL 43

Revert to Snapshot: Files vs. Vvols

flatfilemain.vmdk main.vmdk

redologsnap.vmdk

snap.vmdkVVolID86

VVolID42

backoutVVolID243

SnapShotdata

CONFIDENTIAL 44

vSphere Virtual Volumes Is An Industry-wide Initiative

46

Multiple Available

Now

Unique capabilities

And Many More…

39 Partners in the Program

CONFIDENTIAL

47

The Benefits of vSphere Virtual VolumesA More Efficient Operational Model For External Storage

Improves Resource Utilization

• Increase capacity utilization.• Eliminate overprovisioning• Reduce management overhead

• Eliminate inefficient handoffs between VI and Storage Admin

• Faster storage provisioning through automation

• Simplified change management through flexible consumption

• Self-service provisioning via cloud automation tools

Simplifies Storage Operations

• Leverage native array-based capabilities

• Fine control at the VM level• Dynamic configuration on the fly• Ensure compliance through policy

enforcement using automation

Simplifies Delivery of Service Levels

CONFIDENTIAL

Virtual VolumesTechnical Deep Dive

Ken Werneburg, VMware, IncPatrick Dirks, VMware, Inc

STO4649

#STO4649

Backup

Operation ScenariosOffload, Migrations, Snapshots

53

Migration Scenario: with VAAI vs Virtual Volumes

VASA API Compatible Array

2

vSphere

VMFS VVOLs

1

vSphere Admins

Migrate VM from VMFS toVVOL datastore

software data mover implementation

CONFIDENTIAL 54

55

vSphere Virtual Volumes Offload Operations

vSphere

VMFS VVOLs

vSphere Admins

offload to array

offload to array

Virtual Machine Operation Offloaded• Virtual Machine provisioning

• Virtual Machine deletes

• Virtual Machine full clones

• Virtual Machine Linked Clones

• Virtual Machine Snapshots

• Storage vMotion (Powered off VMs)

CONFIDENTIAL

VM operation from VVOLs container to different VVOLs container

vSphere Admins

vSphere

1

2

offload to array

3

Fully VAAI & VASA APIs Compatible Array

SC-A SC-B

always attempt defaultoperation with VASA API primitives

If default operations fails VAAI API primitives are used

VM operation from VVOLs container to different VVOLs container

vSphere Admins

vmkernel data mover uses

VAAI primitives for cloning operation

vSphere

1

2

offload to array

3

Fully VAAI & VASA APIs Compatible Array

SC-A SC-B

default 1fail back 2

vendor native clone utilized with VASA primitives

cloning operation

VAAI vs VVOLs

CONFIDENTIAL 58