2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Application-Level Benchmarking with SPEC SFS® 2014
Nick Principe – EMC Vernon Miller – IBM
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Agenda
Why application-level benchmarking? What is application-level benchmarking? The SPEC SFS 2014 Workloads
Reporting SFS 2014 Results Testing a “Storage Solution” Ramifications of application-level benchmarking
Concepts Real-life examples
Key takeaways Q&A
2
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
SPEC Standard Performance Evaluation Corporation
The Standard Performance Evaluation Corporation (SPEC) is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops benchmark suites and also reviews and publishes submitted results from member organizations and other benchmark licensees
www.spec.org SPEC and SPEC SFS are registered trademarks of the
Standard Performance Evaluation Corporation. Additional product and service names mentioned herein may be the trademarks of their respective owners.
3
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Why application-level benchmarking?
The focus of the SFS benchmark has changed in SFS 2014 Load is now generated at the application level The aim is to measure the storage
performance of the environment as a whole We call this the “Storage Solution”
Vendors now have the flexibility to configure the benchmark to match their environment Put the bottleneck where you want to show value
4
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Why application-level benchmarking?
Increased flexibility addresses the market Complexity of storage solutions Diversity of architectures and protocols Fairness to all implementations
Any other approach is no longer appropriate for industry-standard benchmarking of storage solutions
5
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
What is Application-Level Benchmarking?
Prior to SFS 2014, benchmark generated its own NFS or SMB traffic Bypassed load generator operating system Focus: performance of monolithic NAS server
SFS 2014 uses native OS calls to generate application-level load Data and metadata ops processed by OS Focus: storage performance of the
environment as a whole (Storage Solution) 6
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
The SPEC SFS 2014 Workloads
DATABASE Simulates OLTP database consolidation Measured in # of concurrent DATABASES
SWBUILD Simulates large software project compilation Measured in # of concurrent BUILDS
VDA Simulates acquisition of streaming data Measured in # of concurrent STREAMS
VDI Simulates heavy steady-state VDI workload Measured in # of concurrent DESKTOPS 7
For more details, see: - SDC 2014 presentation: SPEC SFS 2014: An Under-the-Hood Review - The SPEC SFS 2014 website http://www.spec.org/sfs2014
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Reporting SFS 2014 Results
Disclosure of SPEC SFS 2014 results must meet the requirements of
SPEC SFS 2014 License SPEC SFS 2014 Run and Reporting Rules SPEC Fair Use Rules
Submission to SPEC for review encouraged Certain information is required to be disclosed
Do not use this presentation as a guide for public disclosure of SFS 2014 results
Created for education under auspices of SPEC using “generic” environments 8
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Testing a “Storage Solution”
More attention to benchmark configuration required You must put the bottleneck in the right place
Understanding the whole system, from load generator to the disks, is a requirement
For publication, more configuration details must be recorded and disclosed
SFS 2014 is still a storage benchmark There is no attempt to simulate compute load
9
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Concepts
Your load generators matter Any config detail can affect performance
Storage connectivity, OS version, patch level, memory, client count, tuning parameters
You can test anything that provides a file API to an application Traditional NAS server, block storage with a
file system on load generators, hyper-converged solutions, a single server with storage 10
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Concepts
With great power comes great responsibility Understand where your bottleneck is
SFS 2014 allows great flexibility in load placement as the workload scales
Getting this right is the key to getting the performance you expect Likely you want to spread load as evenly as possible
across ALL resources as the benchmark ramps up load
The key config parameter in SFS 2014? CLIENT_MOUNTPOINTS 11
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Concepts
Measuring performance at multiple levels of the solution under test is key to understanding your solution’s performance and bottlenecks
SFS 2014 reports application-level performance Other statistics that are helpful to collect:
Storage array statistics, NAS server statistics, Hypervisor statistics, LG OS statistics
You may see different performance at the different levels Each layer of the solution under test may change the
workload 12
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 1)
Configured an environment for testing Midrange Storage Array
FC drives, FC frontend
Large Windows Server 2012 R2 NAS Server FC backend, 48 cores, 256GB memory, 10GbE frontend
20 Windows 8.1 VMs; 10 physical servers FC or 10GbE SMB3 connectivity, 2 cores, 2 GB memory
Testing was done in two ways Via SMB3 shares from NAS server (10GbE) Via local E: drive, FC LUN via RDM to VM (FC) 13
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 1)
14
ESXi Server 1
10 GbE Switch
NAS Server
Storage Controller
SMB Server Measurements
Disk Measurements
Network Measurements
Application Measurements Win VM
Win VM
ESXi Server 2 Win VM
Win VM
ESXi Server 10
Win VM
Win VM
8 Gb FC Switch
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 2) Configured an environment for testing Midrange Storage Array
SAS drives, FC frontend
4 Node Distributed Filesystem 2 nodes with FC backend, DDR IB cluster network 2 nodes acting as NFS server, 10 GbE frontend
2 NFS Clients RHEL 6.5, 10GbE connectivity, 4 cores, 32 GB memory
Testing was done in two ways Via NFSv3 exports from NAS server (10GbE) Via local filesystem: single namespace on 2 nodes 15
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 2)
16
NFS Client NFS Client
10 GbE Switch
DDR IB Switch
Frontend Node Frontend Node
Backend Node Backend Node
Storage Controller
Load generators for NFSv3 tests
Load generators for local filesystem tests
NFS Server Measurements
Disk Measurements
Network Measurements
Application Measurements
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 2)
17
0
50,000
100,000
150,000
200,000
250,000
300,000
350,000
400,000
450,000
500,000
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
Kilo
byte
s/Se
cond
Streams
VDA - NFS
App Read App Write Net Recv
Net Send Disk Read Disk Write
0
50,000
100,000
150,000
200,000
250,000
300,000
350,000
400,000
450,000
500,000
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
Kilo
byte
s/Se
cond
Streams
VDA - Cluster FS
App Read App Write
Disk Read Disk Write
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 1)
18
0
50,000
100,000
150,000
200,000
250,000
300,000
350,000
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75
Kib
ibyt
es/s
ec
Streams
VDA - SMB
App Read App Write Net Recv
Net Send Disk Read Disk Write
0
50,000
100,000
150,000
200,000
250,000
300,000
350,000
400,000
450,000
10 20 30 40 50 60 70 80 90 100
Kib
ibyt
es/s
ec
Streams
VDA - Local FS
App Read App Write
Disk Read Disk Write
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 2)
19
0
200
400
600
800
1,000
1,200
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
Ope
rati
ons/
Seco
nd
Streams
VDA - NFS
App NFS Total NFS Data NFS Meta Disk
0
200
400
600
800
1,000
1,200
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
Ope
rati
ons/
Seco
nd
Streams
VDA - Cluster FS
App Disk
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 1)
20
0
100
200
300
400
500
600
700
800
900
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75
Ope
rati
ons/
sec
Streams
VDA - SMB
App SMB Total SMB Data SMB Meta Disk
0
200
400
600
800
1,000
1,200
1,400
1,600
1,800
2,000
10 20 30 40 50 60 70 80 90 100
Ope
rati
ons/
sec
Streams
VDA - Local FS
App Disk
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples
21
0
10
20
30
40
50
60
70
80
90
100
0 200 400 600 800 1,000 1,200
Ave
rage
Res
pons
e T
ime
(ms)
Achieved Ops/Sec
VDA - Env 2
NFSv3 Local
0
50
100
150
200
250
0 200 400 600 800 1,000 1,200
Ave
rage
Res
pons
e T
ime
(ms)
Achieved Ops/sec
VDA - Env 1
SMB3 Local
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 2)
22
0
20,000
40,000
60,000
80,000
100,000
120,000
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40
Kilo
byte
s/Se
cond
Desktops
VDI - NFS
App Read App Write Net Recv
Net Send Disk Read Disk Write
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40
Kilo
byte
s/Se
cond
Desktops
VDI - Cluster FS
App Read App Write
Disk Read Disk Write
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 1)
23
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
1 2 3 4 5 6 7 8 9 10 11 12
Kib
ibyt
es/s
ec
Desktops
VDI - Local FS
App Read App Write
Disk Read Disk Write
0
5,000
10,000
15,000
20,000
25,000
30,000
1 2 3 4 5 6 7 8 9 10 11
Kib
ibyt
es/s
ec
Desktops
VDI - SMB
App Read App Write Net Recv
Net Send Disk Read Disk Write
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 2)
24
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40
Ope
rati
ons/
Seco
nd
Desktops
VDI - NFS
App NFS Total NFS Data NFS Meta Disk
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40
Ope
rati
ons/
Seco
nd
Desktops
VDI - Cluster FS
App Disk
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 1)
25
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
1 2 3 4 5 6 7 8 9 10 11
Ope
rati
ons/
sec
Desktops
VDI - SMB
App SMB Total SMB Data SMB Meta Disk
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
1 2 3 4 5 6 7 8 9 10 11 12
Ope
rati
ons/
sec
Desktops
VDI - Local FS
App Disk
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples
26
0
2
4
6
8
10
12
14
0 2,000 4,000 6,000 8,000 10,000
Ave
rage
Res
pons
e T
ime
(ms)
Achieved Ops/Sec
VDI - Env 2
NFSv3 Local
0
1
2
3
4
5
6
0 500 1,000 1,500 2,000 2,500 3,000
Ave
rage
Res
pons
e T
ime
(ms)
Achieved Ops/sec
VDI - Env 1
SMB3 Local
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 2)
27
0
20,000
40,000
60,000
80,000
100,000
120,000
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40
Kilo
byte
s/Se
cond
Desktops
VDI - NFS (Read Bandwidth)
App Read
Disk - Original
Disk - Tuning #1
Disk - Tuning #2
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 2)
28
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Kilo
byte
s/Se
cond
Builds
SWBUILD - NFS
App Read App Write Net Recv
Net Send Disk Read Disk Write
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Kilo
byte
s/Se
cond
Builds
SWBUILD - Cluster FS
App Read App Write
Disk Read Disk Write
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 1)
29
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
2 4 6 8 10 12 14 16 18 20
Kib
ibyt
es/s
ec
Builds
SWBUILD - Local FS
App Read App Write
Disk Read Disk Write
0
10,000
20,000
30,000
40,000
50,000
60,000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Kib
ibyt
es/s
ec
Builds
SWBUILD - SMB
App Read App Write Net Recv
Net Send Disk Read Disk Write
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 2)
30
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Ope
rati
ons/
Seco
nd
Builds
SWBUILD - NFS
App NFS Total NFS Data NFS Meta Disk
0
2,000
4,000
6,000
8,000
10,000
12,000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Ope
rati
ons/
Seco
nd
Builds
SWBUILD - Cluster FS
App Disk
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples (Env 1)
31
0
2,000
4,000
6,000
8,000
10,000
12,000
2 4 6 8 10 12 14 16 18 20
Ope
rati
ons/
sec
Builds
SWBUILD - Local FS
App Disk
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Ope
rati
ons/
sec
Builds
SWBUILD - SMB
App SMB Total SMB Data SMB Meta Disk
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Ramifications of application-level benchmarking: Real-life Examples
32
0
2
4
6
8
10
12
0 2,000 4,000 6,000 8,000 10,000 12,000
Ave
rage
Res
pons
e T
ime
(ms)
Achieved Ops/Sec
SWBUILD - Env 2
NFSv3 Local
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0 2,000 4,000 6,000 8,000 10,000 12,000
Ave
rage
Res
pons
e T
ime
(ms)
Achieved Ops/sec
SWBUILD - Env 1
SMB3 Local
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
33
Ramifications of application-level benchmarking: Real-life Examples (Env 2)
Percent of I/O
Client Cache Hit
NAS Server Cache Hit Cluster/Disk Cache Hit
Disk Access
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Key takeaways
The SPEC SFS 2014 is an application-level benchmark that tests the storage performance of an entire storage solution
Understanding the storage solution under test and bottleneck placement are keys to getting “what you expect” from your storage solution
The application-level benchmarking provided by SFS 2014 allows testing of a much wider array of products and storage solutions
34
2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.
Q & A
Any questions?
Thank you for attending! Please remember to submit feedback!
35
Top Related