Tech Preview: RDMA and for publication Next-Gen …#vmworld Tech Preview: RDMA and Next-Gen Storage...

#vmworld

Tech Preview: RDMA and Next-Gen Storage

Technologies for vSANBiswapati Bhattacharjee, VMware, Inc.Srinath Premachandran, VMware, Inc.

HCI2476BU

#HCI2476BU

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion

Disclaimer

2©2018 VMware, Inc.

This presentation may contain product features orfunctionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion

Agenda


vSAN Adoption and Overview

Server Side Technology Evolution

vSAN Vision on RDMA

vSAN Vision on Next-Gen Storage Technology

Q & A

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


vSAN Adoptionand Overview

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


HCI is the Fastest Growing Storage SegmentHCI is replacing traditional storage in the enterprise

Source: Wikibon Server SAN Research Project, 2016 Source: IDC Worldwide Quarterly Enterprise Storage Systems Tracker

Traditional Storage Systems Shipped

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion

vSAN Is Adding 100 Customers Per Week

vSAN Customer Adoption

>15,000 CustomersMore than top 3 competitors combined

>70% YoY bookings growthVMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Data Criticality

Number of Workloads

Business Critical

Applications

There Is a Clear HCI Adoption Curve

ROBO Management

ClustersVDI

Test/Dev

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Brief Overview of VMware vSAN

Runs on any standardx86 server

Pools SSDs/HDDs intoa shared datastore

Delivers enterprise-grade security, scale and performance

Managed throughper-VM storage policies

Deeply integrated withthe VMware stack

vSAN Storage

Managed by vCenter

vSphere vSAN

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Server SideTechnology Evolution

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion

Hardware Segmentation of vSAN ReadyNode Platforms

Emerging MarketExisting Market

StorageDense

Compute Intense

Composable Infrastructure

Edge / IoT

Archiving, Video streaming, Analytics Web apps, VDI

Data warehouse, Log aggregation ROBO

Rack Servers Blade Servers Composable Infrastructure Edge Computing

HPE – ApolloCisco S-series

HPE BL-SeriesCisco M4, M5 – BL series, Dell FX2 ….

HPE Synergy (Independent

Compute &Storage)…

Cisco – E seriesDell – R640 …

General Purpose

Sample Use cases BCA, Database, VDI, DR

HardwareType Rack Servers

Supported HardwareExamples

Dell – R740, 640HPE – DL 360/380Cisco – C220/240

Lenovo ThinkSystems,Supermicro, Fujitsu ……VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


How Has Compute Evolved in x86 Servers?

Yesterday Today

Key Trends

• CPU core density is increasing significantly – 12 cores popular today, 24 cores in next 2-3 years

• CPU speeds are becoming faster• FPGA and CPU offloads for higher efficiency• Beefy single socket servers are gaining traction

Dual Socket(12 – 16 Cores)

Dual Socket(8 Cores)

Pro

cess

ing

Po

wer Dual Socket

(24 – 32 Cores)Single Socket

(32 – 64 cores)

FPGA

Intel Cascade Lake(PMEM Enablement)

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


How Has Storage Evolved in x86 Servers?

Yesterday Today

Memory&

Storage

Storage

NAND based Intel NVMe

(250K IOPS)

Intel Optane NVMe SSD(500K IOPS)

DDR4 based Persistent Storage Class Memory(Million IOPS,

Nanosecond latency)

Key Trends• Flash adoption is

transitioning from SAS /SATA to NVMe

• 500K-750K IOPS is expected to be the norm for NVMe devices in next 2 yrs

(50K-80K IOPS)

Key Trends• New Persistent

memory (storage class memory) started appearing in near future

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Today’s Flash Device Landscape

Flash Devices Endurance (DWPD)

ExpectedPerformance

(IOPS) Latency (µs)Capacity

Range (TB)

Persistent Memory In Thousands In Millions In Nanoseconds 0.64 – 0.512

High End NVMe(Intel Optane, P48xx) 30 500K+ 10 0.4 – 0.6

Performance NVMe(Intel Cliffdale, P46xx) 3-5 ~350K 40 1.2 – 1.4

Mainstream SAS(SAS SSD) 2-3 ~250K 60 1.6 – 4.0

Mainstream NVMe (Intel Cliffdale, P45xx) 1-2 ~200K 70 1.6 – 2.0

SATA/Essentials NVMe(Intel Cliffdale, P45x1)


How Has Network Evolved in x86 Servers?

Yesterday Today

10G/25G/40G Network

1G/10G Network

100G/200G Network

Key Trends• 10G/25G/40G is mainstream now

• RDMA enabled (RoCE) NIC is becoming standardized

• 100G projected to be next generation NIC in next 3-5 years

• NVMe-oF technology opens up new storage deployment models choice for HCI

RDMA Enabled

NIC

Net

wo

rk S

pee

d

NVMe-oF

FPGA

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Server Side Technology Evolution & Economics Is Driving HCI

Scale Up Architecture(Traditional Storage Arrays)

Fiber Channel

Scal

e-U

p

SAS SAS SAS SAS

SASExpress

SASExpress

SASExpress

SASExpress

Ethernet

Fibre Channel

Scale Out Architecture(Modern Storage)

Ethernet

Scale-Out

HCI brings in compute, storage and network together with single management interface

Server Side Economics for HCI:1. Falling price of Flash2. Faster & Powerful CPUs &

flash (NVMe) in x86 server3. Unified Management VMw

orld 201

8 Conten

t: Not for

publicat

ion or dis

tribution


This Evolution Enables New Apps and Capabilities for HCI

In MemoryDatabase Big Data

StreamingApplications

Heavy Sequential Write Application

Real-time Processing

Application Require Faster

Reboot, HA

High Performance Computing

Others…..

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


vSAN RDMA Visionand Demo

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


What Is RDMA?

In computing, remote direct memory access (RDMA) is a direct memory access from the memory of one computer into that of another without involving either one's operating system. This permits high-throughput, low-latency networking, which is especially useful in massively parallel computer clusters.

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Above bullets points are different scenarios where RDMA potentially could benefit. However experiments need to be done in each area before anything planned. There

is no commitment on any of the above items and timeline yet to be determined

vSAN Vision on RDMA

Potential RDMA Benefits:

Improved application performance

Better VM Consolidation

Speeding up cloning & vMotion operations

Faster metadata updates across vSAN hosts on the network

Further improve resync/rebuild/rebalance time

NVMe-oF technology enablement

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


These experiments are not final nor been executed on release build. Early experiments to explore how beneficial RDMA could potentially be in vSAN

Few Early Tech Preview Experiments of vSAN over RDMA

Mellanox ConnectX 5 100 Gbps (MCX16A-CCAT)

Mellanox Switch SN2700

2 Disk Groups in Each Node• All NVMe (Cache + Capacity)• 1 Cache + 1 Capacity

2 * 14 core Intel® Xeon® E5-2697 @ 2.60 GHz

512 GB Host Memory

Default VM Storage Policy (FTT=1)

NO Additional vSAN Data Services Turned ON

HCI Bench Tool

Early Exploratory Work with Partner Mellanox®

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Typical Enterprise Workload IO Profile

Typical Workload Block Size Sample Applications

70/30 RW (100% Random) 4K - 16KDatabase applications running• OLTP - SQL/Oracle• Exchange

100% Write (Sequential) 64K - 256K

• Transactional Logs• Big Data• Analytics• Streaming Media• Backup VDI Admin Operations (Clone,

Compose etc.)

100% Write (Random) 32K - 64K • Web Application

100% Read (Sequential) 64K - 256K • Restore from backup/archive

30/70 RW (100% Random) 4K - 16K • VDI Steady State• VDI Boot Storm

70/30 RW (100% Random) 24K-32K • SharePoint

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Exploratory Testing Throughput Numbers for RDMA and TCP/IP over vSAN


No. of Threads

RDMA TCP/IP

6 1545.42 1278.19

6 2995.17 2533.35

6 5287.53 4614.59

6 11331.56 9706.61

6 10813.35 9422.67

6 12716.34 10715.93

1 9099.93 8558.96

70/30 Read/Write ProfileThroughput

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Exploratory Testing IOPS Numbers for RDMA and TCP/IP over vSAN


No. of Threads

RDMA TCP/IP

6 395630.9 327213.3

6 383381.5 324271.8

6 338403.7 295334.1

6 181305 155307.3

6 86506.7 75381.2

6 50865.4 42863.8

1 18199.8 17118

70/30 Read/Write Profile

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Exploratory Testing Latency Numbers for RDMA and TCP/IP over vSAN


No. of Threads

RDMA TCP/IP

6 2.4 2.9

6 2.4 2.9

6 2.8 3.2

6 5.6 5.6

6 11.5 12.9

6 18.9 22.3

1 8.7 9.2

70/30 Read/Write Profile

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


RDMA shows higher throughput, IOPS and lower latency compared to TCP/IP

Across IO profile and sizes, RDMA potentially doing more than TCP/IP

With more IOPS, Throughput; potentially application performance and VM consolidation can be increased

Takeaways of RDMA over vSAN Exploration

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


vSAN Vision on Next-Gen Storage Technology

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


vSAN Architectural Overview

Source: https://download3.vmware.com/vcat/vmw-vcloud-architecture-toolkit-spv1-webworks/index.html#page/Storage%20and%20Availability/Architecting%20VMware%20vSAN%206.2/Architecting%20Virtual%20SAN%206.2.2.008.html

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion

https://download3.vmware.com/vcat/vmw-vcloud-architecture-toolkit-spv1-webworks/index.html#page/Storage%20and%20Availability/Architecting%20VMware%20vSAN%206.2/Architecting%20Virtual%20SAN%206.2.2.008.html


PMEM/ NVDIMM Overview

Current trend on memory and storage technology convergence creates new generation of devices

Persistent Memory (PMEM/NVDIMM) brings in storage to memory slots

These devices are persistent like storage and at the same time fast like memory. Provides load/store memory semantics

Has memory like performance

Block or byte-addressable. Extremely low latency and high endurance

Projected capacity ranges from 256G to Terabytes

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Next-Gen Storage Devices Vision for vSAN

Overview*• Application performance needs are driving

evolution of fast storage such as NVMe & PMEM at a decreasing price point

• As device capacities are getting exponentially larger, might consider contributing capacity from the caching device

• Even further, a single tier design could be beneficial for efficient use of emerging devices like PMEMs for performance-orientate workloads

Potential Benefits• As PMEM capacity and price points roll out, it

could be a play for just metadata or mix of metadata and capacity or just capacity as well

• To leverage ops/performance of these devices having knobs to control space efficiency at a VM granularity could be offered

CONCEPT ONLY

*This diagram is for conceptual depiction only. There is no commitment or timeline on whether vSAN would evolve in this direction from its current

implementation

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Concept Demo: NVMe-oF for VMware vSAN

Compute Only ESXi NodesJBOF Resource Pool - A

JBOF Resource Pool - BCompute Only ESXi NodesNVMe-oF

NVMe-oF potential benefits for vSAN

• Scale storage and compute independently

• Retain the simplicity of HCI management for provisioning and disaggregation workflows

Storage Node – Intel®Storage System

RAF1000JSPCompute Node – Standard x86 Server Hardware

RDMA Over Converged Ethernet (RoCE)

Please visit VMware Advance Technology booth at VMVillage to know more about this demo VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Server side technology evolution & economics is driving HCI

Application performance needs are driving evolution of fast storage and network such as NVMe & PMEM, RDMA at a decreasing price point.

vSAN is committed to explore new technologies (PMEM, RDMA, NVMe, NVMe-oF and others) and adopt as they deem fit in HCI

Key Takeaways

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Visit the HCI Zone in the Solutions Exchange

#vSAN #HCIZone#vSANfan

Experience Solutions Powered by vSANEdge

Video Analytics

CloudPKS with containerized MySQL

CoreSAP HANA on vSAN

DataStax

Espresso Bar vSpeaking Podcast Daily Prizes

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion


Get Ahead of the Curve – vSAN Private Beta

vSAN Data Protection

Native enterprise-grade protection

vSAN File Services

Expanding vSAN beyond block storage

Cloud Native Storage

Persistent storage for containers

Sign up at http://www.vmware.com/go/vsan-beta

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion

PLEASE FILL OUTYOUR SURVEY.Take a survey and enter a drawingfor a VMware company store gift card.

#vmworld #HCI2476BU

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion

THANK YOU!

#vmworld #HCI2476BU

VMworld

2018 Co

ntent: No

t for pub

lication o

r distribu

tion

Tech Preview: RDMA and Next-Gen Storage Technologies for vSANSlide Number 2Slide Number 3vSAN Adoption�and OverviewHCI is the Fastest Growing Storage SegmentvSAN Is Adding 100 Customers Per WeekThere Is a Clear HCI Adoption CurveBrief Overview of VMware vSANServer Side�Technology EvolutionHardware Segmentation of vSAN ReadyNode PlatformsHow Has Compute Evolved in x86 Servers?How Has Storage Evolved in x86 Servers? Today’s Flash Device LandscapeHow Has Network Evolved in x86 Servers?Server Side Technology Evolution & Economics Is Driving HCIThis Evolution Enables New Apps and Capabilities for HCIvSAN RDMA Vision�and DemoWhat Is RDMA?vSAN Vision on RDMAFew Early Tech Preview Experiments of vSAN over RDMATypical Enterprise Workload IO ProfileExploratory Testing Throughput Numbers for RDMA and TCP/IP over vSANExploratory Testing IOPS Numbers for RDMA and TCP/IP over vSANExploratory Testing Latency Numbers for RDMA and TCP/IP over vSANTakeaways of RDMA over vSAN ExplorationvSAN Vision on Next-Gen Storage TechnologyvSAN Architectural OverviewPMEM/ NVDIMM OverviewNext-Gen Storage Devices Vision for vSANConcept Demo: NVMe-oF for VMware vSANSlide Number 31Key TakeawaysVisit the HCI Zone in the Solutions ExchangeGet Ahead of the Curve – vSAN Private BetaSlide Number 35Slide Number 36

Tech Preview: RDMA and for publication Next-Gen …#vmworld Tech Preview: RDMA and Next-Gen Storage...

Documents

Transcript of Tech Preview: RDMA and for publication Next-Gen …#vmworld Tech Preview: RDMA and Next-Gen Storage...