Ken Cantrell / NetApp Mark Rogov / EMC July 30, 2015.

STORAGE PERFORMANCE BENCHMARKING: INTRODUCTION AND FUNDAMENTALS

Ken Cantrell / NetApp

Mark Rogov / EMC

July 30, 2015

© 2015 Storage Networking Industry Association. All Rights Reserved. 2

About The Speakers

Mark RogovEMC

Systems Engineer

@rogovmark

Dr. J MetzCisco

R&D Engineer

@drjmetz

Ken CantrellNetApp

Manager Perf Engineering

@kencantrelljr

© 2015 Storage Networking Industry Association. All Rights Reserved.

SNIA Legal Notice

The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted.

Member companies and individual members may use this material in presentations and literature under the following conditions:

Any slide or slides used must be reproduced in their entirety without modification

The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.

This presentation is a project of the SNIA Education Committee.

Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney.

The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information.

NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.

3


Storage Performance Benchmarking

SYSTEM UNDER TEST

METRICS ANDTERMINOLOGY

FILECOMPONENTS

BLOCKCOMPONENTS

WORKLOADDEFINITIONS



SYSTEM UNDER TEST


FILECOMPONENTS

BLOCKCOMPONENTS

WORKLOADDEFINITIONS

TODAY FUTURE WEBCASTS

INTRO

BLOCK

FILE GRAPH


BUSINESSRUN PRODUCTION

APPLICATIONS

6

Three Great Buckets Of Testing

DATABASECOMPUTE FARM

VDI

INTRO

BLOCK

FILE GRAPH



APPLICATIONS

BAKE-OFFSSINGLE OBSCURE METRIC

7


UN*X DDIOMETERIOZONE

VDBENCH


VDI

INTRO

BLOCK

FILE GRAPH



APPLICATIONS


8



VDBENCH


VDI

INTRO

BLOCK

FILE GRAPH


BENCHMARKSBALANCED

PROFESSIONAL APPROACH


APPLICATIONS


9



VDBENCH


VDI

LOGIN VSIVMMARK

SPEC SFSSPC1

INTRO

BLOCK

FILE GRAPH


Block / SAN Terminology

IO IOIOIOIOIOIO BLOCK / SANPROTOCOLFC/iSCSI/FCoE

Targ

et

Init

iato

r

INTRO

BLOCK

FILE GRAPH



IO IOIOIOIOIOIO

Targ

et

BLOCK / SANPROTOCOLFC/iSCSI/FCoE

IO OPERATION

AKA TYPE:READWRITE

Init

iato

r

INTRO

BLOCK

FILE GRAPH


Block / SAN Operations

Source: http://www.t10.org/cgi-bin/ac.pl?t=f&f=sbc4r07c.pdf Graphic: http://www.wordle.net/create

INTRO

BLOCK

FILE GRAPH



IO IOIOIOIOIOIO

Targ

et


IO OPERATION

AKA TYPE:READWRITE

Init

iato

r

IOPSCOUNT EVERY IO OPERATIONPER SECOND

THROUGHPUTSAME AS IOPS

INTRO

BLOCK

FILE GRAPH


Where To Measure IOPS?

HOSTSSTORAGE

CONTROLLERDISK DRIVES

INTRO

BLOCK

FILE GRAPH



HOSTSSTORAGE


500

400

300

200

100

FRONT END

1500

INTRO

BLOCK

FILE GRAPH



HOSTSSTORAGE


500

400

300

200

1001000

1000

200

IDLE

15002200

FRONT ENDBACK END

INTRO

BLOCK

FILE GRAPH



IO IOIOIOIOIOIO

Targ

et


IO OPERATION

AKA TYPE:READWRITE

Init

iato

r


MB/SPAYLOAD SUM

OF EVERY OPERATION

PER SECOND

BANDWIDTHSAME AS MB/S

INTRO

BLOCK

FILE GRAPH



10004

TERABYTESTB

10003

GIGABYTESGB

MB/s Vs. MiB/s

MEGABYTESMB 10002

10001

KILOBYTESKB

10244

TEBIBYTESTiB

18

10243

GIBIBYTESGiB

10242MEBIBYTES

MiB

10241

KIBIBYTESKiB

10%

7%

5%

2%

DECIMAL BINARY

INTRO

BLOCK

FILE GRAPH



IO IOIOIOIOIOIO

Targ

et


IO OPERATION

AKA TYPE:READWRITE

IO SIZESIZE OF DATA

PAYLOADIN AN IO

OPERATION

Init

iato

r


MB/SPAYLOAD SUM

OF EVERY OPERATION

PER SECOND


INTRO

BLOCK

FILE GRAPH



MB/s (or MiB/s), IO Size, And IOPS

4,096 40,960 409,6000

50

100

150

200

250

300

350

400

450

500

MiB/s For Different IO Sizes

512B1KiB4KiB8KiB16KiB

IOPS

MiB

/s

MiB / s = 𝐼𝑂𝑆𝑖𝑧𝑒∗𝐼𝑂𝑃𝑆KEEPING IOPS

CONSTANT, LARGER I/O

SIZES PRODUCE

GREATER MiB/S

512B

1KiB

4KiB

INTRO

BLOCK

FILE GRAPH


MB/s (or MiB/s), IO Size, And IOPS

4,096 40,960 409,6000

50

100

150

200

250

300

350

400

450

500

MiB/s For Different IO Sizes

512B1KiB4KiB8KiB16KiB

IOPS

MiB

/s

MiB / s = 𝐼𝑂𝑆𝑖𝑧𝑒∗𝐼𝑂𝑃𝑆AS IO SIZE

INCREASES IOPS

DECREASE FOR THE SAME

MiB/S

16KiB 8KiB 4KiB

INTRO

BLOCK

FILE GRAPH


IO SIZESIZE OF DATA

PAYLOADIN AN IO

OPERATION

22


IO IOIOIOIOIOIO

Targ

et


IO OPERATION

AKA TYPE:READWRITE

Init

iato

r


MB/SPAYLOAD SUM

OF EVERY OPERATION

PER SECOND


RESPONSETIME

TIME TARGETTAKES TO REPLY

TO AN IO

LATENCYSAME AS

RESPONSE TIME

INTRO

BLOCK

FILE GRAPH



Aggregate Property Of Response Time

DISK DRIVESSTORAGE

CONTROLLER HOSTS

INTRO

BLOCK

FILE GRAPH



DISK DRIVESSTORAGE

CONTROLLER HOSTS

WRITE ≥ 1ms1ms

INTRO

BLOCK

FILE GRAPH



DISK DRIVESSTORAGE

CONTROLLER HOSTS

READ

WRITE

≥ 1.5ms

≥ 1ms1ms

1ms

0.5ms

INTRO

BLOCK

FILE GRAPH



DISK DRIVESSTORAGE

CONTROLLER HOSTS

READ

WRITE

≥ 1.5ms

≥ 1ms

READ

1ms

1ms

≥ 5ms

4ms

0.5ms

INTRO

BLOCK

FILE GRAPH


File / NAS Terminology

OP OPOPOPOPOPOP FILE PROTOCOL NFS/SMB/FTP

Clie

nt

Ser

ver

INTRO

BLOCK

FILE GRAPH




Clie

nt

Ser

ver

FILEOPERATION

SEVERAL DOZENDEPENDING

ONPROTOCOL

INTRO

BLOCK

FILE GRAPH


NFSv4 Operations

Source: https://www.ietf.org/rfc/rfc3530.txtGraphic: http://www.wordle.net/create

INTRO

BLOCK

FILE GRAPH


SMB3 Operations

Source: https://msdn.microsoft.com/en-us/library/cc246482.aspx (use [MS-SMB2].pdf)Graphic: http://www.wordle.net/create

INTRO

BLOCK

FILE GRAPH




Clie

nt

Ser

ver

FILEOPERATION


ONPROTOCOL

OPSCOUNT EVERY

PROTOCOLOPERATION

PER SECOND

INTRO

BLOCK

FILE GRAPH




Clie

nt

Ser

ver

FILEOPERATION


ONPROTOCOL

OPSCOUNT EVERY

PROTOCOLOPERATION

PER SECOND

MB/SPAYLOAD SUM

OF EVERY OPERATION

PER SECOND

THROUGHPUTSAME AS MB/S


INTRO

BLOCK

FILE GRAPH




Clie

nt

Ser

ver

FILEOPERATION


ONPROTOCOL

OPSCOUNT EVERY

PROTOCOLOPERATION

PER SECOND

MB/SPAYLOAD SUM

OF EVERY OPERATION

PER SECOND



INTRO

BLOCK

FILE GRAPH




Clie

nt

Ser

ver

FILEOPERATION


ONPROTOCOL

OPSCOUNT EVERY

PROTOCOLOPERATION

PER SECOND

MB/SPAYLOAD SUM

OF EVERY OPERATION

PER SECOND



RESPONSETIME


TO AN IO

LATENCYSAME AS

RESPONSE TIME

INTRO

BLOCK

FILE GRAPH


Difference Between OPS and MB/s (MiB/s)Two Otherwise Identical Runs With Different NFS Operations Mix (64 KiB IO Size)

70

30

High MetaData Operations

Stat Read

30

70

High Data Opera-tions

Stat Read

INTRO

BLOCK

FILE GRAPH



70

30


Stat Read

30

70


Stat Read0

200

400

600

800

1000

1200

1400

1600

1800

2000

0

5

10

15

20

25

30

35

40

45

OPS MB/s

High MetaDataOperations

High DataOperations

INTRO

BLOCK

FILE GRAPH



70

30


Stat Read

30

70


Stat Read0

200

400

600

800

1000

1200

1400

1600

1800

2000

0

5

10

15

20

25

30

35

40

45

OPS Read MiB/s Stat MiB/s

High MetaDataOperations

High DataOperations

INTRO

BLOCK

FILE GRAPH


Graph Fun Or Why “It Depends” Is Relevant?

4000 5000 6000 7000 8000 9000 10000 11000 12000 13000 14000

Operations per second, OPS

5,000 10,000

INTRO

BLOCK

FILE GRAPH



0 2000 4000 6000 8000 10000 12000 140000

2

4

6

8

10

12

1

1

Response time per OPS

system 1 system 2

Operations per second, OPS

Re

spo

nse

tim

e, m

s

INTRO

BLOCK

FILE GRAPH



1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000

5

10

15

20

25Response time per OPS

System 1 System 2

Operations per sec, OPS

Re

spo

nse

tim

e,

ms

INTRO

BLOCK

FILE GRAPH



1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000

5

10

15

20


System 1 System 2


Re

spo

nse

tim

e,

ms

INTRO

BLOCK

FILE GRAPH


MAXIMUM PERFORMANCE

NEED


1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000

5

10

15

20


System 1 System 2


Re

spo

nse

tim

e,

ms

INTRO

BLOCK

FILE GRAPH


MINIMUM PERFORMANCE REQUIREMENT


1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000

5

10

15

20


System 1 System 2


Re

spo

nse

tim

e,

ms

INTRO

BLOCK

FILE GRAPH


ACCEPTABLE LATENCY BAND


1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000

5

10

15

20


System 1 System 2


Re

spo

nse

tim

e,

ms

INTRO

BLOCK

FILE GRAPH


ACCEPTABLE LATENCY BAND

Variance

1000 1550 2100 2650 3200 3750 4300 4850 5400 5950 6500 7050 7600 8150 8700 9250 9800 103501090011450120000

5

10

15

20


System 1 System 2


Re

spo

nse

tim

e,

ms

INTRO

BLOCK

FILE GRAPH


Other Metrics To Consider

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000

5

10

15

20


System 1 System 2


Re

spo

nse

tim

e,

ms

$2/OP

$10/OP

$1.1/OP

$5.5/OP

INTRO

BLOCK

FILE GRAPH



1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000

5

10

15

20


System 1 System 2


Re

spo

nse

tim

e,

ms

$1.1/OP

$5.5/OP$25K/U

$1K/U

$40/W

$140/W

}@9000 OPS @ 17 ms

}@9000 OPS @ 10 ms

INTRO

BLOCK

FILE GRAPH



1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000

5

10

15

20


System 1 System 2


Re

spo

nse

tim

e,

ms

$1.1/OP

$5.5/OP$25K/U

$1K/U

$40/W

$140/W

}@9000 OPS @ 17 ms

}@9000 OPS @ 10 ms

CONTEXTMAKES

METRICSMATTER

INTRO

BLOCK

FILE GRAPH


Summary

System 1 System 2

GRAPH FUN

CONTEXTMAKES

METRICSMATTER

OPSCOUNT EVERY

PROTOCOLOPERATION

PER SECOND

MB/SPAYLOAD SUM

OF EVERY OPERATION

PER SECOND

TERMINOLOGY


RESPONSETIME


TO AN IO

INTRO

BLOCK

FILE GRAPH



SYSTEM UNDER TEST


FILECOMPONENTS

BLOCKCOMPONENTS

WORKLOADDEFINITIONS

TODAY FUTURE WEBCASTS


After This Webcast

This Webcast and a PDF of the slides will be posted to the SNIA Ethernet Storage Forum (ESF) website and available on-demand

http://www.snia.org/forums/esf/knowledge/webcasts

A full Q&A from this webcast, including answers to questions we couldn't get to today, will be posted to the SNIA-ESF blog

http://sniaesfblog.org/

Follow us on Twitter @SNIAESF

Next Webcast - October 21st

“Storage Performance Benchmarking: Part 2”

http://www.snia.org/forums/esf/knowledge/webcasts

http://sniaesfblog.org/

https://twitter.com/SNIAESF

QUESTIONS?

THANK YOU!

Ken Cantrell / NetApp Mark Rogov / EMC July 30, 2015.

Documents

Transcript of Ken Cantrell / NetApp Mark Rogov / EMC July 30, 2015.