Ken Cantrell / NetApp Mark Rogov / EMC J Metz / Cisco October 21, 2015.

STORAGE PERFORMANCE BENCHMARKING:PART 2 – SOLUTION UNDER TEST

Ken Cantrell / NetAppMark Rogov / EMC

J Metz / CiscoOctober 21, 2015

© 2015 Storage Networking Industry Association. All Rights Reserved.

SNIA Legal NoticeThe material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions:

Any slide or slides used must be reproduced in their entirety without modificationThe SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.

This presentation is a project of the SNIA Education Committee.Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney.The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information.

NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.

2

© 2015 Storage Networking Industry Association. All Rights Reserved.© 2015 Storage Networking Industry Association. All Rights Reserved.

About The Speakers

3

Mark RogovEMC

Systems Engineer@rogovmark

Dr. J MetzCisco

R&D Engineer@drjmetz

Ken CantrellNetApp

Manager Perf Engineering@kencantrelljr


SPEC SFS® 2014See http://spec.org/sfs2014/ for details

SWBUILD

VDA

VDI

DATABASE

spec©

PROTOCOL SUPPORT• ANY TRANSPORT PROTOCOL

• TCP/IP, FC, ISCSI, ETC.• ANY NETWORK PROTOCOL

• NFS, SMB, ETC.• USES POSIX FILE OPERATIONS

METRICS• BUSINESS METRICS COUNTS

• SOFTWARE BUILDS, VIRTUAL DESKTOPS, VIDEO STREAMS, DATABASES

• RESPONSE TIME

http://spec.org/sfs2014/

http://spec.org/sfs2014/

© 2015 Storage Networking Industry Association. All Rights Reserved. 5

Storage Performance Benchmarking

SOLUTIONUNDER TEST

METRICS ANDTERMINOLOGY

FILECOMPONENTS

BLOCKCOMPONENTS

WORKLOADDEFINITIONS




FILECOMPONENTS

BLOCKCOMPONENTS

WORKLOADDEFINITIONS

TODAY FUTURE WEBCASTSJULY 30, 2015

SOLUTIONUNDER TEST

ENDSUTINTRO CACHE MOLE HERO MiB/S


Metrics and Terminology ReviewPart 1: http://www.snia.org/forums/esf/knowledge/webcasts (Both PDF and PPT available)

System 1 System 2

GRAPH FUN

CONTEXTMAKES

METRICSMATTER

OPSCOUNT EVERY

PROTOCOLOPERATION

PER SECOND

MB/SPAYLOAD SUM

OF EVERY OPERATION

PER SECOND

TERMINOLOGY

IOPSCOUNT EVERY IO OPERATIONPER SECOND

RESPONSETIME

TIME TARGETTAKES TO REPLY

TO AN IO

ENDMiB/SHEROMOLECACHESUTINTRO

http://www.snia.org/forums/esf/knowledge/webcasts



Last Time: Which is Better?

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000

5

10

15

20

25Response time per OPS

System 1 System 2

Operations per sec, OPS

Res

pons

e tim

e, m

s



Last Time: Which is Better?

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000

5

10

15

20


System 1 System 2


Res

pons

e tim

e, m

s

CONTEXTMAKES

METRICSMATTER



This Time: Why Are They Different?

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000

5

10

15

20


System 1 System 2


Res

pons

e tim

e, m

s



What is a Solution Under Test (SUT)?

HYPERVISOR

VM VM

HYPERVISOR

VM VM

Increasing Complexity



SUT Layers

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT



Front- and Back-End Basic Difference

2KiB IOs

FRONT ENDBACK END

8KiB IOs

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT



RUN ON

Is a Workload / Application a Part of the SUT?

WORKLOADS

HYPERVISOR

VM VM

HYPERVISOR

VM VM

SOLUTIONS UNDER TEST



Which Elements in the SUT Affect Performance?

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT



What is Common Between All Of The Components?

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

PROCESSING ELEMENTS (CPU, ASIC)

MEMORY / CACHE



Answer To Ken’s Interview Question

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

CACHE

CLOUD

IOIO



3 Principles To Improve Performance

DO LESS WORKDO WORK FASTER INCREASE PARALLELISM



Latency Or “Whack A Mole Game”

0

2

4

6

8

10

12

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

7200 RPM

1000x

Res

pons

e Ti

me

(ms)



Disk Bottleneck

0

2

4

6

8

10

12

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

DISK BOUNDaka DISK BOTTLENECK

Res

pons

e Ti

me

(ms)

1000x

7200 RPM



Why Are SSDs So Compelling?

25

DRAM SSD HDD0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

Res

pons

e Ti

me

(ns)

Random Write Latency

11ms

0.1ms


25ns


Why Are SSDs So Compelling?

26

DRAM SSD HDD

108

107

106

105

104

103

102

101

100

Res

pons

e Ti

me

(ns)

Random Write Latency

11ms

0.1ms


25ns

100X


Change SUT: Upgrade With SSDs

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

SSDWAS 12ms

Res

pons

e Ti

me

(ms)

6x

DO WORK FASTER



Change SUT: Upgrade With SSDs

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

Res

pons

e Ti

me

(ms)

DO WORK FASTER


CONTROLLERBOUND

WAS 12ms


Controller Bottleneck

00.05

0.10.15

0.20.25

0.30.35

0.40.45

0.5

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

VENDOR

Res

pons

e Ti

me

(ms)

~1.8x

CLIENTBOUND

WAS 0.7ms

DO WORK FASTER



Bottlenecks Always Exist

CLIENTS /HOSTS

STORAGECONTROLLER

00.05

0.10.15

0.20.25

0.30.35

0.40.45

0.5

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

BOTTLENECKMOVES

Res

pons

e Ti

me

(ms)

DO WORK FASTER


WAS 12ms


Back to the Original Problem …

0

2

4

6

8

10

12

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

Res

pons

e Ti

me

(ms)

DO LESS WORK



Avoid Slow Parts And Generate “Hero Numbers”

0

2

4

6

8

10

12

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

ADDMORE CACHE

DON’T USE THIS

Res

pons

e Ti

me

(ms)

DO LESS WORK



Use More Cache

-1.11022302462516E-16

0.0999999999999999

0.2

0.3

0.4

0.5

0.6

0.7

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

Res

pons

e Ti

me

(ms)

WAS 12ms

CONTROLLERBOUND

DISKLATENCY IS STILL 11MS

BUTMASKED

DO LESS WORK



Caching Isn’t Just For Slow Drives

3,000 MILES

~70MS LATENCY

CLOUDSTORAGE

STORAGECONTROLLER

BACK-ENDCONNECT


DO LESS WORK


Latency Isn’t Everything

00.05

0.10.15

0.20.25

0.30.35

0.40.45

0.5

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

STILL A BOTTLENECKHOW?

Res

pons

e Ti

me

(ms)



MiB/s Limits

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

NOT ENOUGH DISKS

MiB

/s

“MORE”IS

BETTER


INCREASE PARALLELISM


MiB/s Limits

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT

MiB

/s

“MORE”IS

BETTER



BOTTLENECKMOVES


Network Bound Or Design Problem

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT



BOTTLENECK


Network Bound Or Design Problem

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT




Which SUT Component Matters?

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECT



Which Component Matters?

CLIENTS /HOSTS

STORAGECONTROLLER

PHYSICALSTORAGE

FRONT-ENDCONNECT

BACK-ENDCONNECTTHE SLOWESTSUT COMPONENT

MATTERSMOST



Solution Under Test Review

DISK BOUND

CLIENTBOUND

SLOW COMPONENTMATTERS MOST

BOTTLENECKSALWAYS EXIST

3 PERFORMANCEPRINCIPLES


DO LESSWORK

DO WORKFASTER





FILECOMPONENTS

BLOCKCOMPONENTS

WORKLOADDEFINITIONS

TODAY FUTURE WEBCASTSJULY 30, 2015

SOLUTIONUNDER TEST



After This Webcast

A PDF and a PPT of the slides for this and all previous parts of this Webcast series will be posted to the SNIA Ethernet Storage Forum (ESF) website and available on-demand

PPT and PDF: http://www.snia.org/forums/esf/knowledge/webcasts Storage Performance Benchmarking: Part 1 Recording: https://www.brighttalk.com/webcast/663/164323

A full Q&A from this webcast, including answers to questions we couldn't get to today, will be posted to the SNIA-ESF blog

http://sniaesfblog.org/

Follow us on Twitter @SNIAESF, @RogovMark, @KenCantrellJr, @DrJMetzNext Webcast – First Quarter 2016

“Storage Performance Benchmarking: Part 3”



https://www.brighttalk.com/webcast/663/164323

http://sniaesfblog.org/

QUESTIONS?

THANK YOU!

Ken Cantrell / NetApp Mark Rogov / EMC J Metz / Cisco October 21, 2015.

Documents

Transcript of Ken Cantrell / NetApp Mark Rogov / EMC J Metz / Cisco October 21, 2015.