SER2413BE NVMe: What Is It? An Interface? A Protocol? A or ... · scale-out storage Multi-tenancy,...

Sudhanshu (Suds) Jain & Adrian Marinescu

SER2413BE

#VMworld #SER2413BE

NVMe: What Is It? An Interface? A Protocol? A New Drive Technology? An Industry Revolution?

VMworld 2017 Content: Not fo

r publication or distri

bution

Agenda

#SER2413BU CONFIDENTIAL 2

1 Why do we need faster storage?

2 Where is the holdup?

3 What is NVMe?

4 NVMe in vSphere

5 Future EvolutionVMworld 2017 Content: Not fo


bution

Rise of the Third Platform

3

Transforming the World of Applications

Highly parallel processing on very large set of data

Need to deliver business value in

real-time

Low latency for mission critical transactions

Deliver risk and opportunity for future

Business Intelligence

Real-time analytics

Transactional Processing

Machine

Learning

Predictive Analytics

Artificial

Intelligence

Hyper-scale Datacenter

$5.3 trillion by 2020: IDC

#SER2413BU CONFIDENTIAL



bution

Key Infrastructure Challenges


EXPLOSIVE GROWTH OF

DATA

NEAR REALTIME

HIGHLY DISTRIBUTED

Towards Data-centric Future!



bution

Explosive Growth of Data


Stressing your storage and network benchmarks

Source: Intel IDF 2016

Cloudera

ZookeeperVMworld 2017 Content: N

ot for publicatio

n or distribution

Business Value of Data Diminish Fast


Time = $$$

Source: http://jtonedm.com/2012/11/21/decision-latecy-revisited/

What we do when data is changing Act fast to gain business value



bution

Highly Distributed

7

Build, deploy and manage interconnected, collaborative workloads

Big Data Cloud-Native App Multi-tier App

Deploy multiple workloads with strong

demand for inter-VM traffic

Optimize data delivery to applications

Adaptive storage tier critical for improved

application performance

Distributed Storage

Virtual

SAN




bution

Composable Infrastructure


Disaggregation of Infrastructure

Source: http://myeventagenda.com/sessions/0B9F4191-1C29-408A-8B61-65D7520025A8/14/5#sessionID=1484

Performance Curev

All infrastructure delivered as a service

Hyper-scalable to keep up with business demands

Resources automatically tuned to application workloads



bution

Disruption In Making

9

Future is here now

Source: http://www.theregister.co.uk/2016/09/05/wikibon_server_san_takeover/

Technology Trends

CPU densities

continues to

increase

Hi-Density Flash and

NVDIMM reducing cost

of data access

High speed interconnect

can deliver any to any

fast storage


Application demands for data and key technology trends are disrupting the entire SAN & DAS ecosystem!



bution

Agenda




3 What is NVMe?

4 NVMe in vSphere



bution

Anatomy of Data AccessWhere Is the Holdup?

11

Source: http://www.snia.org/sites/default/education/tutorials/2012/fall/solid/AnilVasudeva_NVMe_NextGen_SSD%20Interface-r1-nc1.pdf

* - Typical/Worst case considerationFor long period of time, innovation in silicon (processor

and memory) industry far outpaced the disk I/O speedup




bution

Anatomy of Data Access


Let’s Look Closer on the Various Delays

Source: https://www.snia.org/sites/default/files/SDC15_presentations/gen_sessions/JimHandy_The_Long-Term_Future_PrintVersion.pdf

Various Delays?

How does various

Technology perform?



bution

Agenda




3 What is NVMe?

4 NVMe in vSphere



bution

Non Volatile Memory Express: A New Protocol

• NVM Express defines a host controller interface and command set for use with non-volatile memory (NVM)

• Key differentiators

– Streamlined Commands; lighter weight I/O Stack

– Direct access to CPU; lower latency

– Massive parallelism; higher throughput

– Faster media

– Scalable performance

• Key Innovations

– http://www.nvmexpress.org

– http://www.snia.org

– https://www.jedec.org/

– http://www.onfi.org


NVMe is architected for NVM

Source: http://www.datacenterjournal.com/answer-nvme-questions/



bution

http://www.nvmexpress.org/

http://www.snia.org/

https://www.jedec.org/

http://www.onfi.org/

NVMe: A New Drive Technology


NVMe need new controller and faster media to deliver on its promise!

Source: http://www.snia.org/sites/default/files/files2/files2/contrib_articles2014/S4-S7%20SNIA%20Europe%20v3%20Single%20Pages.pdf

The head has to move to the right place and wait for the right block to

come around

Ah, no need to wait for the blocks to come to the right place – picking

blocks when you need them, one at a time

Getting multiple blocks of information all at once! It is fast

HDD SSD NVMe



bution

http://www.snia.org/sites/default/files/files2/files2/contrib_articles2014/S4-S7 SNIA Europe v3 Single Pages.pdf

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4Q1 Q2 Q3 Q4

NVM Express, Inc. Roadmap

2014 2015 2017 Future direction

NVM

Express

2016

Released NVMe Planned NVMe Specification releases

• Sanitize

• Streams

• Virtualization

NVMe 1.2.1 May’16 NVMe (next)*

• IO Determinism

• Persistent Cntlr Mem Buffer

• Multipathing

NVMe-MI

NVMe

over

Fabrics

NVMe-MI 1.0 Nov’15 NVMe-MI 1.1*

• SES

• NVMe-MI In-band

• Native Enclosure Mgmt

• Transport and protocol

• RDMA binding

NVMe-oF 1.0 May’16 NVMe-oF (next)*

• Enhanced Discovery

• Authentication

• TCP Transport

• Namespace Management

• Controller Memory Buffer

• Host Memory Buffer

• Live Firmware Update

NVMe 1.2 – Nov ‘14

• Out-of-band management

• Device discovery

• Health & temp monitoring

• Firmware Update

* Subject to change

NVMe 1.3 May’17



bution


NVMe: Revolutionizing the SAN Architecture

17

NVMe Over Fabric

Scalability

IOPS

LatencyBandwidth NVMe

NVMf

The value of data is based on how fast it can

be accessed and processed

Faster storage access enables cost reduction

through consolidation

Independently scale storage and compute

infrastructures

Data resilience – Mitigate the “server stranded

storage” (with DAS storage) problem

Optimized data delivery to applications

Efficiency of shared storage with no

compromise

Unlocking the Reach of Data



bution

Storage Industry is Pivoting Towards NVMe

18

NVMe Adoption and Considerations

Currently using

Planning to use within 12 months

Not using and no plans

Drivers to NVMe Adoption

https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2017/20170809_S202A_Burgener.pdf

0% 10% 20% 30% 40% 50% 60%

Better scalability to meet ourperformance requirements over

time

Better cost/performance ($/IOPS)

Higher throughput or bandwidth

Increased storage density (TB/U)

Lower latency

48%

PCIe or NVMe Flash




bution

Market Trends – High Growth Rate for NVMe


It is not a coincidence!

Source: snia.org & IDC

NVMe Drive Revenues growing at 47%

CAGR; expected to hit $7.3B by 2020

Overall revenue from NVMe and NVMe

related product and services is many fold

Key factors driving adaption

• Higher capacity drive and lower cost

• Availability of higher endurance flash media

• Migration from SLC to MLC

Overall revenue of flash media has already exceeded HDD; NVMe is the main growth engine

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

2016 2017 2018 2019 2020

Component Revenues: NVMe, HDD, SSD

NVMe HDD SSD

$7.3B

$11.8B

$16.2B



bution

Agenda




3 What is NVMe?

4 NVMe in vSphere



bution

vSphere Platform

21

Telco & HPC

Cloud

VSAN

NSX

Ultra-low latency, packet

intensive workloads,

scale-out storage

Multi-tenancy, zero-trust,

Specialized hardware and

customization

Low latency, high-IOPS &

predictable performance

High packet rate, fast packet

manipulation, hardware offloads,

All-IP networks

FOUNDATIONAL TO SDDC

vSphere Platform

Transforming World of Enterprise

Scale, HPC-like and Highly

Distributed




bution

vSphere Driver Architecture Evolution

22

Ran

do

m, IP

MI

RDMA

RoC

E, iW

AR

P

SW

FCoE

FC

oE

Driv

er

Nativ

e S

AS

/FC

Driv

er

Nativ

e N

IC D

rive

r

Nativ

e G

rap

hic

s D

rive

r

Additional Partner Native Drivers

vmklinux

SA

S/F

C D

rive

r

NIC

Driv

er

SATA

Library

SA

TA

Driv

er

SW FCoE

FC

oE

Driv

er

Feature focus area

VMK APIVMK APIVMK API

Legacy support



bution

Software-Defined Infrastructure


NVMe/Flash is Key Focus Area

• vSphere Flash Use-Cases: (KB 2145210)

• Host swap cache

• Regular Datastore

• vSphere Flash Read Cache (aka Virtual Flash)

• VSphere ESXi Boot Disk

• VSphere ESXi Coredump device

• VSphere ESXi Logging device

• Virtual SAN (VSAN)

• Open Source Effort to build for faster innovation and the broad

ecosystem

• Open ecosystem for certification and support

Flash and NVMe



bution

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2145210

vSphere Storage I/O Stack


Built for Best of the Breed Fabric Solutions!

Virtual VolumesVirtual SAN VMFS

Monitor

SSD PCIe NVMeDIMM/NVDIMM PCI SSD

LSI PVSCSI vNVMe

HBANICCPU

HDD

LSI Driver PCSCI DriverNative NVMe

Driver

Guest

I/O Driver NVMe

RDMANIC SAS/SCSI

iSCSIFCoE

NFS

Physical Hardware

S

C

S

I

S

T

A

C

K

N

V

M

e

S

T

A

C

K

FC/SAS

NVMeOverFabric

OS Stack

vRDMA

Para Virtualized

RDMA

ESXi

vmxnet3

VDS

NIOCScheduler

Memory AllocatorSIOC

Vmxnet3 Driver

T

C

P

/

I

P

S

T

A

C

K



bution

NVM Express Evolution and vSphere

25

Journey towards All NVMe Stack

• Queueing Interface• NVM Command Set• Admin Command Set• End- to- end Protection (DIF /DIX )• Security• Physical Region Pages ( PRPs )

2009 2011 20122010 2013 2014

NVMe 1. 0 ReleasedMarch 1 , 2011

NVMeTechnical Work Begins

•Implementation and Reporting Refinements

NVMe 1. 2 ReleasedNovember 3 , 2014

•Name Space Management

•Controller Memory Buffer


• Ground Work for NVMe Management

2015

• General Scatter Gather Lists (SGLs)

• Multi-Path I/O & Namespace Sharing• Reservations• Autonomous Power Transitions

During Idle

NVMe 1. 1 ReleasedOctober 11 , 2012

2016

•Defines extension to NVMe, for non PCI

NVMe Over Fabric 1. 0 ReleasedJune 5 , 2016

•Primary focus on RDMA

•Compatible to FC-NVMe (INCITS 540)


• Ground Work for NVMe Management

vSphere 5.5

• Introduce first async NVMe driver 1.0e

• Launch IOVP cert program for NVMe

vSphere 6.0

• Introduce first inbox NVMe driver

• Bring broader ecosystem support

vSphere 6.5

• vNVMe• Optimized

performance for NVMe driver

Future Direction

• Multiple name spaces, Queues

• NVMe Over Fabric• End-to-end NVMe

Stack



bution

NVMe Driver Ecosystem

• Available as part of base ESXi image from vSphere 6.0 onwards

– Faster innovation with async release of VMware NVMe driver

• VMware led vSphere NVMe Open Source Driver project to encourage ecosystem to innovate

– https://github.com/vmware/nvme

• Broad NVMe Ecosystem on VMware NVMe Driver https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io

– Close to 300 third party NVMe devices certified on VMware NVMe driver

• Also available for download (async) VMware ESXi 5.5 nvme 1.2.0.27-4vmw NVMe Driver for PCI Express based Solid-State Drives

#SER2413BU CONFIDENTIAL26



bution

https://github.com/vmware/nvme

https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io

https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESX55-VMWARE-NVME-12027-4&productId=353

NVMe Focus @VMware

27

Build Your Infrastructure with Confidence!

• Reduced serialization

• Locality improvements

• vNVMe Adaption layer

• Multiple completion worlds support in

NVMe

Core Stack

• Boot (UEFI)

• Firmware Update

• End-to-end protection

• Deallocate/TRIM/Unmap

• 4K

• SMART, Planned hot-remove

Driver

• NVMe 1.0e spec

• Hot-plug support

• VM orchestration

Virtual Devices

• Optimized stack - Highly parallel

execution for single path local NVMe

devices

• Reach target of 90%+ performance of

device spec

• Performance enhancements

• Extended CLI/UI

• Name space management

• Async event error handling

• Enhance diagnostic logs

• Performance improvements

• Async mode support

• unmap support

vSphere 6.5 Tech Preview

• Next Generation Storage Stack

with ultra-high IOPS

• End-to-end NVMe Stack

• NVMe Multi-pathing

• NVMe Over Fabric

• Multiple fabric option

• SR-IOV

• Sanitize

• I/O Determinism

• Rev the specification

• Parallel execution @backend

• 4K Support

• Scatter-gather support

• Interrupt coalescing

Future Direction



bution

Key Challenges for NVMe Based Architecture


NVMe Require Fresh Approach for Software Stack Designs

Parallel Execution

Elevator / C-Scan forced stack to be serialized

File systems are also not optimized for concurrent

NUMA architectures pose additional challenges for locality

Interrupts distribution and queue selection

IO scheduling

Performance

More processing power needed to saturate NVMe device

IOPS increase by almost order of magnitude

IO overhead reduced by 2-3x

Queuing == increases latencyVMworld 2017 Content: N

ot for publicatio

n or distribution

Application Application Application

File System

IO

Scheduler

Block

storage

Driver

Implie

d s

erializ

ation

SSD

Source: https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2013/20130812_PreConfD_Marks.pdf

Processing

resources

100 KIOPS

Typical Storage Stack

29#SER2413BU CONFIDENTIAL



bution

NVMe Focus @VMware

30






NVMe

Core Stack

• Boot (UEFI)

• Firmware Update



• 4K


Driver

• NVMe 1.0e spec



Virtual Devices



devices


device spec


• Extended CLI






• unmap support








• SR-IOV



• 4K Support



Future Direction




bution

ESXi Storage Stack Enhancements for NVMe Performance Boost


0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1 2 4 8

Th

rou

gh

pu

t (I

OP

S)

# Workers

• Hardware:

– Intel® Xeon® E5-2687W v3 @3.10GHz (10 cores + HT)

– 64 GB RAM

– NVM Express* 1M IOPS @ 4K Reads

• Software:

– vSphere* 6.0U2 vs. Future prototype

– 1 VM, 8 VCPU, Windows* 2012, 4 VMDK eager-zeroed

– IOMeter:

• 4K seq reads, 64 OIOs per worker, even distribution of workers to VMDK

vSphere today

vSphere prototype

Source: SDC Conference 2016 https://www.snia.org/sites/default/files/SDC/2016/presentations/performance/Chagam_Dutta_Rajaopal_Bridging_the_Gap_Between_NVMe_SSD_Performance_and_Scale_Out_Software.pdf



bution

vSphere Virtualized I/O Stack


vSphere Offers Rich Set of I/O Virtualization

Virtual VolumesVirtual SAN VMFS

Monitor

NVMe SSD PCI SSD

LSI PVSCSI vNVMe

Interconnect like

IB and OPANIC

HDD

NIC Driver PCSCI DriverNative NVMe

Driver

Guest

I/O Driver NVMe

RDMANIC SAS/SCSI

iSCSIFCoE

NFS

Physical Hardware

P

C

I

FC/SAS

NVMeOverFabric

OS Stack

vRDMA

Para Virtualized

RDMA

ESXi

vmxnet3

VDS

NIOCScheduler

Memory AllocatorSIOC

I/O Device Driver

D

i

r

e

c

t

P

a

s

s

-

t

h

r

o

u

g

h

PCIe GPU/GPGPU FPGA

Hardware Accelerations

e.g QAT

Device

Emulation

Para-

Virtualized

Device

S

C

S

I

S

T

A

C

K

N

V

M

e

S

T

A

C

K

T

C

P

/

I

P

S

T

A

C

K

NSX

DatapathvSAN

Datapath



bution

Virtual NVMe


High Performance Guest Block I/O

New in

vSphere

6.5

Feature:

• NVMe 1.0e Device Emulation

• Hot add/remove support

• Integrated with VM orchestration

• Multi-Q support – 16Queues, total 4K depth

Benefits:

• Improved application performance, better

IOPS and latency numbers

• Leverage native NVMe stack in various

Guest OS including Linux and Windows

• Reduced in-guest I/O stack overhead



bution

Virtual NVMe


Rich Set of Features & Potential!

Implemented

34



bution

NVMe Focus @VMware

35






NVMe

Core Stack

• Boot (UEFI)

• Firmware Update



• 4K


Driver

• NVMe 1.0e spec



Virtual Devices



devices


device spec


• Extended CLI/UI






• unmap support








• SR-IOV

• Sanitize

• I/O Determinism



• 4K Support



Future Direction




bution

NVMe Focus @VMware

36

Summary





NVMe

Core Stack

• Boot (UEFI)

• Firmware Update



• 4K


Driver

• NVMe 1.0e spec



Virtual Devices



devices


device spec


• Extended CLI/UI






• unmap support








• SR-IOV

• Sanitize

• I/O Determinism



• 4K Support



Future Direction



bution

Agenda




3 What is NVMe?

4 NVMe in vSphere



bution

Byte addressableBlock storage

Storage Evolutionary Path & Key Drivers

38

Building the Stack for Future

System Applications

File System

SCSI

Storage

Architecture

SCSI DriverNVMe/NVMeoF

Driver

NVMe

Storage

Architecture

NVDIMM

Storage

Flash evolution in cost, endurance and

capacity

NVMe as a replacement of SCSI

Composable Infrastructure

NVMe Over Fabric as primary way to

access storage

SAN Evolution towards Server Attached

Storage




bution

Next-Generation Hardware EvolutionNVMe as Caching Tier & Performance Tier

Persistence Tier

Today’s Hardware

NVMeSSD NVDIMM

Evolution of Storage Tier

NVMe

NVDIMM

Caching Tier

NVMe




bution

NVMe Over Fabric: Demo

40

POC

Next Generation SAN is Here!



bution

Key Takeaways

41

Foundation to deploy Software Defined Infrastructure

vSphere and its NVMe ecosystem is constantly innovating

vSphere is ready to power your NVMe infrastructure

NVMe is the choice of technology for your storage needs




bution



bution

SER2413BE NVMe: What Is It? An Interface? A Protocol? A or ... · scale-out storage Multi-tenancy,...

Documents

Transcript of SER2413BE NVMe: What Is It? An Interface? A Protocol? A or ... · scale-out storage Multi-tenancy,...