Post on 01-Jan-2016
STORAGE PERFORMANCE BENCHMARKING: INTRODUCTION AND FUNDAMENTALS
Ken Cantrell / NetApp
Mark Rogov / EMC
July 30, 2015
© 2015 Storage Networking Industry Association. All Rights Reserved. 2
About The Speakers
Mark RogovEMC
Systems Engineer
@rogovmark
Dr. J MetzCisco
R&D Engineer
@drjmetz
Ken CantrellNetApp
Manager Perf Engineering
@kencantrelljr
© 2015 Storage Networking Industry Association. All Rights Reserved.
SNIA Legal Notice
The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted.
Member companies and individual members may use this material in presentations and literature under the following conditions:
Any slide or slides used must be reproduced in their entirety without modification
The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.
This presentation is a project of the SNIA Education Committee.
Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney.
The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information.
NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.
3
© 2015 Storage Networking Industry Association. All Rights Reserved. 4
Storage Performance Benchmarking
SYSTEM UNDER TEST
METRICS ANDTERMINOLOGY
FILECOMPONENTS
BLOCKCOMPONENTS
WORKLOADDEFINITIONS
© 2015 Storage Networking Industry Association. All Rights Reserved. 5
Storage Performance Benchmarking
SYSTEM UNDER TEST
METRICS ANDTERMINOLOGY
FILECOMPONENTS
BLOCKCOMPONENTS
WORKLOADDEFINITIONS
TODAY FUTURE WEBCASTS
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved.
BUSINESSRUN PRODUCTION
APPLICATIONS
6
Three Great Buckets Of Testing
DATABASECOMPUTE FARM
VDI
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved.
BUSINESSRUN PRODUCTION
APPLICATIONS
BAKE-OFFSSINGLE OBSCURE METRIC
7
Three Great Buckets Of Testing
UN*X DDIOMETERIOZONE
VDBENCH
DATABASECOMPUTE FARM
VDI
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved.
BUSINESSRUN PRODUCTION
APPLICATIONS
BAKE-OFFSSINGLE OBSCURE METRIC
8
Three Great Buckets Of Testing
UN*X DDIOMETERIOZONE
VDBENCH
DATABASECOMPUTE FARM
VDI
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved.
BENCHMARKSBALANCED
PROFESSIONAL APPROACH
BUSINESSRUN PRODUCTION
APPLICATIONS
BAKE-OFFSSINGLE OBSCURE METRIC
9
Three Great Buckets Of Testing
UN*X DDIOMETERIOZONE
VDBENCH
DATABASECOMPUTE FARM
VDI
LOGIN VSIVMMARK
SPEC SFSSPC1
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 10
Block / SAN Terminology
IO IOIOIOIOIOIO BLOCK / SANPROTOCOLFC/iSCSI/FCoE
Targ
et
Init
iato
r
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 11
Block / SAN Terminology
IO IOIOIOIOIOIO
Targ
et
BLOCK / SANPROTOCOLFC/iSCSI/FCoE
IO OPERATION
AKA TYPE:READWRITE
Init
iato
r
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 12
Block / SAN Operations
Source: http://www.t10.org/cgi-bin/ac.pl?t=f&f=sbc4r07c.pdf Graphic: http://www.wordle.net/create
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 13
Block / SAN Terminology
IO IOIOIOIOIOIO
Targ
et
BLOCK / SANPROTOCOLFC/iSCSI/FCoE
IO OPERATION
AKA TYPE:READWRITE
Init
iato
r
IOPSCOUNT EVERY IO OPERATIONPER SECOND
THROUGHPUTSAME AS IOPS
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 14
Where To Measure IOPS?
HOSTSSTORAGE
CONTROLLERDISK DRIVES
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 15
Where To Measure IOPS?
HOSTSSTORAGE
CONTROLLERDISK DRIVES
500
400
300
200
100
FRONT END
1500
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 16
Where To Measure IOPS?
HOSTSSTORAGE
CONTROLLERDISK DRIVES
500
400
300
200
1001000
1000
200
IDLE
15002200
FRONT ENDBACK END
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 17
Block / SAN Terminology
IO IOIOIOIOIOIO
Targ
et
BLOCK / SANPROTOCOLFC/iSCSI/FCoE
IO OPERATION
AKA TYPE:READWRITE
Init
iato
r
IOPSCOUNT EVERY IO OPERATIONPER SECOND
MB/SPAYLOAD SUM
OF EVERY OPERATION
PER SECOND
BANDWIDTHSAME AS MB/S
INTRO
BLOCK
FILE GRAPH
THROUGHPUTSAME AS IOPS
© 2015 Storage Networking Industry Association. All Rights Reserved. 18
10004
TERABYTESTB
10003
GIGABYTESGB
MB/s Vs. MiB/s
MEGABYTESMB 10002
10001
KILOBYTESKB
10244
TEBIBYTESTiB
18
10243
GIBIBYTESGiB
10242MEBIBYTES
MiB
10241
KIBIBYTESKiB
10%
7%
5%
2%
DECIMAL BINARY
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 19
Block / SAN Terminology
IO IOIOIOIOIOIO
Targ
et
BLOCK / SANPROTOCOLFC/iSCSI/FCoE
IO OPERATION
AKA TYPE:READWRITE
IO SIZESIZE OF DATA
PAYLOADIN AN IO
OPERATION
Init
iato
r
IOPSCOUNT EVERY IO OPERATIONPER SECOND
MB/SPAYLOAD SUM
OF EVERY OPERATION
PER SECOND
BANDWIDTHSAME AS MB/S
INTRO
BLOCK
FILE GRAPH
THROUGHPUTSAME AS IOPS
© 2015 Storage Networking Industry Association. All Rights Reserved. 20
MB/s (or MiB/s), IO Size, And IOPS
4,096 40,960 409,6000
50
100
150
200
250
300
350
400
450
500
MiB/s For Different IO Sizes
512B1KiB4KiB8KiB16KiB
IOPS
MiB
/s
MiB / s = 𝐼𝑂𝑆𝑖𝑧𝑒∗𝐼𝑂𝑃𝑆KEEPING IOPS
CONSTANT, LARGER I/O
SIZES PRODUCE
GREATER MiB/S
512B
1KiB
4KiB
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 21
MB/s (or MiB/s), IO Size, And IOPS
4,096 40,960 409,6000
50
100
150
200
250
300
350
400
450
500
MiB/s For Different IO Sizes
512B1KiB4KiB8KiB16KiB
IOPS
MiB
/s
MiB / s = 𝐼𝑂𝑆𝑖𝑧𝑒∗𝐼𝑂𝑃𝑆AS IO SIZE
INCREASES IOPS
DECREASE FOR THE SAME
MiB/S
16KiB 8KiB 4KiB
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved.
IO SIZESIZE OF DATA
PAYLOADIN AN IO
OPERATION
22
Block / SAN Terminology
IO IOIOIOIOIOIO
Targ
et
BLOCK / SANPROTOCOLFC/iSCSI/FCoE
IO OPERATION
AKA TYPE:READWRITE
Init
iato
r
IOPSCOUNT EVERY IO OPERATIONPER SECOND
MB/SPAYLOAD SUM
OF EVERY OPERATION
PER SECOND
BANDWIDTHSAME AS MB/S
RESPONSETIME
TIME TARGETTAKES TO REPLY
TO AN IO
LATENCYSAME AS
RESPONSE TIME
INTRO
BLOCK
FILE GRAPH
THROUGHPUTSAME AS IOPS
© 2015 Storage Networking Industry Association. All Rights Reserved. 23
Aggregate Property Of Response Time
DISK DRIVESSTORAGE
CONTROLLER HOSTS
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 24
Aggregate Property Of Response Time
DISK DRIVESSTORAGE
CONTROLLER HOSTS
WRITE ≥ 1ms1ms
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 25
Aggregate Property Of Response Time
DISK DRIVESSTORAGE
CONTROLLER HOSTS
READ
WRITE
≥ 1.5ms
≥ 1ms1ms
1ms
0.5ms
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 26
Aggregate Property Of Response Time
DISK DRIVESSTORAGE
CONTROLLER HOSTS
READ
WRITE
≥ 1.5ms
≥ 1ms
READ
1ms
1ms
≥ 5ms
4ms
0.5ms
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 27
File / NAS Terminology
OP OPOPOPOPOPOP FILE PROTOCOL NFS/SMB/FTP
Clie
nt
Ser
ver
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 28
File / NAS Terminology
OP OPOPOPOPOPOP FILE PROTOCOL NFS/SMB/FTP
Clie
nt
Ser
ver
FILEOPERATION
SEVERAL DOZENDEPENDING
ONPROTOCOL
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 29
NFSv4 Operations
Source: https://www.ietf.org/rfc/rfc3530.txtGraphic: http://www.wordle.net/create
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 30
SMB3 Operations
Source: https://msdn.microsoft.com/en-us/library/cc246482.aspx (use [MS-SMB2].pdf)Graphic: http://www.wordle.net/create
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 31
File / NAS Terminology
OP OPOPOPOPOPOP FILE PROTOCOL NFS/SMB/FTP
Clie
nt
Ser
ver
FILEOPERATION
SEVERAL DOZENDEPENDING
ONPROTOCOL
OPSCOUNT EVERY
PROTOCOLOPERATION
PER SECOND
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 32
File / NAS Terminology
OP OPOPOPOPOPOP FILE PROTOCOL NFS/SMB/FTP
Clie
nt
Ser
ver
FILEOPERATION
SEVERAL DOZENDEPENDING
ONPROTOCOL
OPSCOUNT EVERY
PROTOCOLOPERATION
PER SECOND
MB/SPAYLOAD SUM
OF EVERY OPERATION
PER SECOND
THROUGHPUTSAME AS MB/S
BANDWIDTHSAME AS MB/S
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 33
File / NAS Terminology
OP OPOPOPOPOPOP FILE PROTOCOL NFS/SMB/FTP
Clie
nt
Ser
ver
FILEOPERATION
SEVERAL DOZENDEPENDING
ONPROTOCOL
OPSCOUNT EVERY
PROTOCOLOPERATION
PER SECOND
MB/SPAYLOAD SUM
OF EVERY OPERATION
PER SECOND
THROUGHPUTSAME AS MB/S
BANDWIDTHSAME AS MB/S
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 34
File / NAS Terminology
OP OPOPOPOPOPOP FILE PROTOCOL NFS/SMB/FTP
Clie
nt
Ser
ver
FILEOPERATION
SEVERAL DOZENDEPENDING
ONPROTOCOL
OPSCOUNT EVERY
PROTOCOLOPERATION
PER SECOND
MB/SPAYLOAD SUM
OF EVERY OPERATION
PER SECOND
THROUGHPUTSAME AS MB/S
BANDWIDTHSAME AS MB/S
RESPONSETIME
TIME TARGETTAKES TO REPLY
TO AN IO
LATENCYSAME AS
RESPONSE TIME
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 35
Difference Between OPS and MB/s (MiB/s)Two Otherwise Identical Runs With Different NFS Operations Mix (64 KiB IO Size)
70
30
High MetaData Operations
Stat Read
30
70
High Data Opera-tions
Stat Read
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 36
Difference Between OPS and MB/s (MiB/s)Two Otherwise Identical Runs With Different NFS Operations Mix (64 KiB IO Size)
70
30
High MetaData Operations
Stat Read
30
70
High Data Opera-tions
Stat Read0
200
400
600
800
1000
1200
1400
1600
1800
2000
0
5
10
15
20
25
30
35
40
45
OPS MB/s
High MetaDataOperations
High DataOperations
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 37
Difference Between OPS and MB/s (MiB/s)Two Otherwise Identical Runs With Different NFS Operations Mix (64 KiB IO Size)
70
30
High MetaData Operations
Stat Read
30
70
High Data Opera-tions
Stat Read0
200
400
600
800
1000
1200
1400
1600
1800
2000
0
5
10
15
20
25
30
35
40
45
OPS Read MiB/s Stat MiB/s
High MetaDataOperations
High DataOperations
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 38
Graph Fun Or Why “It Depends” Is Relevant?
4000 5000 6000 7000 8000 9000 10000 11000 12000 13000 14000
Operations per second, OPS
5,000 10,000
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 39
Graph Fun Or Why “It Depends” Is Relevant?
0 2000 4000 6000 8000 10000 12000 140000
2
4
6
8
10
12
1
1
Response time per OPS
system 1 system 2
Operations per second, OPS
Re
spo
nse
tim
e, m
s
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 40
Graph Fun Or Why “It Depends” Is Relevant?
1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000
5
10
15
20
25Response time per OPS
System 1 System 2
Operations per sec, OPS
Re
spo
nse
tim
e,
ms
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 41
Graph Fun Or Why “It Depends” Is Relevant?
1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000
5
10
15
20
25Response time per OPS
System 1 System 2
Operations per sec, OPS
Re
spo
nse
tim
e,
ms
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 42
MAXIMUM PERFORMANCE
NEED
Graph Fun Or Why “It Depends” Is Relevant?
1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000
5
10
15
20
25Response time per OPS
System 1 System 2
Operations per sec, OPS
Re
spo
nse
tim
e,
ms
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 43
MINIMUM PERFORMANCE REQUIREMENT
Graph Fun Or Why “It Depends” Is Relevant?
1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000
5
10
15
20
25Response time per OPS
System 1 System 2
Operations per sec, OPS
Re
spo
nse
tim
e,
ms
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 44
ACCEPTABLE LATENCY BAND
Graph Fun Or Why “It Depends” Is Relevant?
1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000
5
10
15
20
25Response time per OPS
System 1 System 2
Operations per sec, OPS
Re
spo
nse
tim
e,
ms
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 45
ACCEPTABLE LATENCY BAND
Variance
1000 1550 2100 2650 3200 3750 4300 4850 5400 5950 6500 7050 7600 8150 8700 9250 9800 103501090011450120000
5
10
15
20
25Response time per OPS
System 1 System 2
Operations per sec, OPS
Re
spo
nse
tim
e,
ms
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 46
Other Metrics To Consider
1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000
5
10
15
20
25Response time per OPS
System 1 System 2
Operations per sec, OPS
Re
spo
nse
tim
e,
ms
$2/OP
$10/OP
$1.1/OP
$5.5/OP
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 47
Other Metrics To Consider
1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000
5
10
15
20
25Response time per OPS
System 1 System 2
Operations per sec, OPS
Re
spo
nse
tim
e,
ms
$1.1/OP
$5.5/OP$25K/U
$1K/U
$40/W
$140/W
}@9000 OPS @ 17 ms
}@9000 OPS @ 10 ms
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 48
Other Metrics To Consider
1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 120000
5
10
15
20
25Response time per OPS
System 1 System 2
Operations per sec, OPS
Re
spo
nse
tim
e,
ms
$1.1/OP
$5.5/OP$25K/U
$1K/U
$40/W
$140/W
}@9000 OPS @ 17 ms
}@9000 OPS @ 10 ms
CONTEXTMAKES
METRICSMATTER
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 49
Summary
System 1 System 2
GRAPH FUN
CONTEXTMAKES
METRICSMATTER
OPSCOUNT EVERY
PROTOCOLOPERATION
PER SECOND
MB/SPAYLOAD SUM
OF EVERY OPERATION
PER SECOND
TERMINOLOGY
IOPSCOUNT EVERY IO OPERATIONPER SECOND
RESPONSETIME
TIME TARGETTAKES TO REPLY
TO AN IO
INTRO
BLOCK
FILE GRAPH
© 2015 Storage Networking Industry Association. All Rights Reserved. 50
Storage Performance Benchmarking
SYSTEM UNDER TEST
METRICS ANDTERMINOLOGY
FILECOMPONENTS
BLOCKCOMPONENTS
WORKLOADDEFINITIONS
TODAY FUTURE WEBCASTS
© 2015 Storage Networking Industry Association. All Rights Reserved. 51
After This Webcast
This Webcast and a PDF of the slides will be posted to the SNIA Ethernet Storage Forum (ESF) website and available on-demand
http://www.snia.org/forums/esf/knowledge/webcasts
A full Q&A from this webcast, including answers to questions we couldn't get to today, will be posted to the SNIA-ESF blog
http://sniaesfblog.org/
Follow us on Twitter @SNIAESF
Next Webcast - October 21st
“Storage Performance Benchmarking: Part 2”
QUESTIONS?
THANK YOU!