1

Larry Touchette

DB-3-341 Flash as a Cache on Physical Servers with Fusion-io directCache

V08

Agenda

NetApp Server Cache Partner Program

Fusion-io directCache

− Software Architecture

− Software Behaviors

Video Demo

Performance & Positioning

2

Flash as Cache

3

• Significant performance benefits, but relatively costly

Flash is emerging in server as IOPS tier

• No sharing across servers• No high availability, no integrated data

protection, no disaster recovery• Management complexity

Server flash alone is just fast DAS (Direct Attached Storage)

• Flash to improve performance, network storage to protect and manage data

• Optimal $/GB and $/IOPS

Ideal: Flash as a cache for network storage

5

• NetApp server caching software• Software only, uses supported PCI-e or SSD• Turns server flash into cache for Data ONTAP

Flash Accel

• Validates partner hardware for use with Flash Accel• Validates partner software compatibility with ONTAP• Enables broad use case coverage

Server Caching Partner Alliance

• Resell of hardware and software products• Enables complete server caching solution purchase

from single source

Fusion-io Resale Program

Best of Breed server flash integrated with Data ONTAP VST and data management for simplicity, lower cost and broad workload coverage

NetApp Server Cache Approach

1.0 GA release (February) − Win2008R2 VM / vSphere 5.0 only− NFS or VMFS & RDMs− LSI PCIE, Micron PCIE, Server SSDs

1.1 release (April)− Adds support: VMW HA, vMotion/ DRS− Adds support: for iSCSI inside VM

1.2 release (June)− Adds support: Fusion-io ioDrive PCIE− Adds support: vSphere 5.1

Server Cache Use Case Support Scope

Fusion-io Resell (available now)

− Virtual (ioTurbine) and Physical (directCache) environments

− vSphere 4 and 5 − Windows 2003, 2008/2008R2, Linux− VMware vMotion, HA & DRS support

Other Server Cache Partners (available now)FlashSoft – STEC – LSI

− Support for any VM guest OS− VM Agent or VM Agent-less− Virtual and Physical environments

Partner Feature Comparison on Field Portal

6

NetApp Flash Accel Server Cache Partner Alliance

• When does the cache invalidate data?• Is the cache a write-through or a write back cache?• Does the cache retain data across a reboot etc?

Key criteria include

Data Coherency Implications with Caching

7

010010010010101010101011010101010010100100010010

Storage ArrayPhysical Server

010010010010101010101011010101010010100100010010

Application

Cache

DataCoherency

Why is data coherency important?Caches that don’t recognize changes to the primary data copy (e.g SnapRestore) will cause data corruption issues

8

Fusion-io Bare Metal Cache Solution

vSphere Environments Physical Environments

ioSphere VM Edition ioSphere & CLI

ioMemory VSL (driver)High speed block protocol access to ioMemory

(Independent VSL for Windows, Linux, and ESX)

ioMemory device: ioDrive2

Fusion-io Solution Components

9

Fusion-io directCache

Integrated with ioMemory VSL

Caches block storage only− FC− iSCSI

Does not cache NAS − No CIFS/NFS

10

Physical Server

Application

OS

ioMemory

NetApp®

Storage

VSLDiskController

Multi-Volume Caching Support

One ioMemory device caches multiple LUNs

64 backing stores per ioMemory device

No device partitioningrequired

11

ioMemory Cached data blocks

directCache Transparent Filter Drivers

Provides two filter drivers− Volume filter (NTFS/EXT3-4 layer)

− Disk filter

One filter allowed per backing store

All reads are cached

Writes are committed to disk first, then cached to service later reads

12

File System Stack

File System Filter

File System

Volume Snapshot

directCache

Volume Manager

Disk Class Driver

directCache

Disk Driver

Application

Partition Manager

Windows I/O Path

Volume Stack

Disk Stack

directCache Write-Through Transaction

14

1. Application issues Write IO

2. directCache intercepts and reissues the IO to Backing Store

3. If the block currently exists in the cache, it is invalidated

4. Write transaction received by Backing Store

5. Write transaction is acknowledged by Backing Store

6. VSL issues write to the ioMemory

7. Write transaction is acknowledged by ioMemory

8. Final acknowledgement is sent back to app

File system

Caching Block Device

directCache

ioMemory VSL

Application

4

1

2

5

6

7

8

3

Physical Server

SAN Interop – Note in IMT covers general SAN compatibility when write-through caches are used

− NetApp cluster failover, multipathing, etc…

Certification for SnapDrive for Windows in progress

− IMT compatibility for SnapDrive in progress

− Completed coherency testing matrix at: https://kb.netapp.com/support/index?page=content&id=3013621

Support for Server Side Cache Software

16

directCache Compatibility Testing

17

directCache 1.1.6

SnapDrive® for Windows® 6.4.1(drive letter mount point)

Compatible – requires NetApp SnapRestore Support feature turned on in directCache

SnapDrive for Windows(folder mount point) Not supported – Known Issue*

SnapDrive for UNIX® To be tested

SnapProtect® To be tested

* Data coherency is not automatic when Windows folder mount point is used to connect LUN: KB7010020

Current as of October 2012

Video Demo

Video Demo

− Configuring directCache to enable caching

− Review performance effect after cache warms up

− Configure directCache to disable caching

18

directCache Configuration Recommendations

If using SnapDrive® for Windows®, you must install the directCache NetApp® SnapRestore® Support feature

Do not perform revert of storage (SnapRestore®) at the array manually without invalidating cache

Do not cache LUNs that are heavily written to

− Example: Database logs, SnapInfo LUNs

When using multiple cache devices, spread busiest backing stores across cache devices

19

20

Performance and Positioning

Workload and Configuration Applications

Random read, OLTP, high read:write ratio

− Oracle, SQL, DB2, etc…

Host clustering is supported except for clusters where different nodes simultaneously access the same device

− MSCS, VCS 5.x and above on RHEL, Linux native cluster are supported by directCache

− Oracle RAC is not supported

21

Database Use Cases

22

Application

PrimaryStorage

Cache

Temp on Flash and Cached Primary

TempDB

Application

PrimaryStorage

Cache

Cached Primary

Array Cache / Server Cache Interactions

23

DB Size – 3.6TB, Working Set – 1.2 TB, Cache Sizes – 1 TB, Server Cache hit rate – 70%, Flash Cache hit rate – 55%, R/W Ratio – 90 / 10, Simplified Single Workload Environment, Working Set > Cache. Performance data will change based on system and workload configuration.

No Flash Baseline Array Cache Server Cache (Host) Array Cache & Server Cache0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Total IOPS to NetApp storage IOPS from Server flash NetApp disk util %Configurations

So

luti

on

IO

PS

% U

tili

zati

on

(D

isk

& C

PU

)

41% Gain Over Flash Cache alone @ similar disk utilization levels

Server Cache and Array Cache Interaction

Application data is pulled into server cache

− Cache hits are served from server cache

− Array cache is free for other workloads

Array cache complements server cache

− Offloads metadata IOPS from array disk

− Reduces disk utilization

− Increases CPU utilization

25

Positioning Array Cache and Server Cache

26

Array Cache – should always be positioned for shared storage benefits

− Reduces disk utilization and improves overall array efficiency and performance

− Acceleration maintained through server HA events

− Shared cache for all clients

− Improved acceleration of deduped datasets

Server Cache – should be positioned when

− Need to address application hot spots

Server Caching Sales Kit on fieldportal.netapp.com− Server Caching Partner Alliance FAQ− Fusion-io Resale Collateral− SE Technical Presentations− Product FAQs

External DLs for NetApp Server Caching team− xdl-server-caching@netapp.com

(contains NetApp only)− xdl-fusion-io-presales@netapp.com

(contains Fusion-io & NetApp members)

Resources

27

Summary

28

directCache allows the NetApp Virtual Storage Tier to extend into the server in bare metal configurations

Deploying flash as a cache allows increased performance while maintaining NetApp data management capabilities

Simple administration and configuration allows quick deployment

directCache complements array cache to offload IOPS and increase efficiency

© 2012 NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consent of NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, and Go further, faster, are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such.

CR-2-340 Flash Deployment Strategies: Host-based and Array-Based

CR-2-562 Implementing Host-Side Caching in Virtualized Environments (NetApp Flash Accel)

DB-3-341 Flash as a Cache on Physical Servers with Fusion-io directCache

VI-3-420 Flash as a Cache on Virtual Servers with Fusion-io ioTurbine

Other Server Flash Sessions at Insight

30

CR-2-496 NetApp Flash Pool Deep Dive

VI-2-281 Use Flash Technology to Accelerate Your Virtual Infrastructure

DB-2-548 Oracle Advanced Topics: Flash-This, Flash-That, and DNFS

BD-1-222 Dynamic Disk Pools: Next-Generation RAID Protection for E-Series

CR-3-454 Are SSDs the performance Grail for storage systems?

Other Server Flash Sessions at Insight continued

31

32

Extra Slides

Fusion-io ioSphere (for Bare Metal)

Fusion-io offers two different ioSphere products:

− ioSphere – for bare metal environments using ioMemory and directCache software

− ioSphere VM Edition – management system for ioTurbine software in virtual environments

− These are different products

Customers with valid Fusion-io support contracts are entitled to download ioSphere

33

Fusion-io ioSphere (for Bare Metal)

GUI management and monitoring software for ioMemory devices

Real-time visibility into all deployed ioMemory

Historical performance monitoring

Warranty forecasting

Intuitive Web user interface

34

ioDrive2 ioDrive2 ioDrive2

ioSphere - VM orPhysical Host

Web Client

Fusion-io ioSphere (for Bare Metal)

ioSphere capabilities

35

Interface Web GUI for remote access

AuthenticationLocal users and/or LDAP accounts

Monitoring Multihost ioMemory device monitoring, auto discovery

Alerting SMS/Email, customizable and searchable history

ioMemory Performance metricsLive display and historical reports

Configuration Configure ioMemory devices

Usage warranty forecasting

directCache Terms and Definitions

Important directCache terms and definitions

36

Cache deviceThe ioMemory ioDrive2 device used for flash memory capacity

ioMemory control deviceThe device name for the ioMemory ioDrive2 device. Example: \\.\fct5

Backing store

The primary storage device to be cached. A device name, drive letter, or mount point. Example: PhysicalDrive3, F:\, or C:\MSSQL\tempdb

Bind To configure a backing store to be cached

UnbindTo remove a backing store from a cache device and drop cache contents

directCache Installation Overview

Installation high-level steps

1. Install ioDrive2 card in host

2. Install ioMemory VSL

3. Install directCache software

4. If using SDW install the directCache NetApp ®

SnapRestore® Support feature

5. Install ioSphere software (optional) on different windows host

6. Configure storage for caching

37

Configuring directCache

1. Identify control device name for ioMemory

38

Device name\\.\fct5

C:\Users\Administrator> fio-status

Found 1 ioMemory device in this systemFusion-io driver version: 3.1.3 build 27

Adapter: Single Controller AdapterFusion-io ioDrive2 785GB, Product Number:F00-001-

785G...Connected ioMemory modules:fct5: Product Number:F00-001-785G-CS-0001, SN:1210D0132

fct5 Attached as 'fct5' (block device)ioDrive2 Adapter Controller, Product Number:F00-001-

785G...PCI:05:00.0Firmware v6.0.0, rev 107004 Public785.00 GBytes block device size

Configuring directCache

2. Nondisruptively bind cache device to backing store

39

Note: binding a system/boot drive is not supported by directCache.

C:\Users\Administrator>dc-bind C:\MSSQL\tpce_tempdb \\.\fct5About to create binding using backing store 'C:\MSSQL\tpce_tempdb’ caching device '\\.\fct5’ write-through mode

WARNING: Binding will destroy any existing data on the caching device \\.\fct5!

Do you wish to continue [y/n]? yBinding: [====================] (100%) /Successfully bound backing store 'C:\MSSQL\tpce_tempdb’ with cache device '\\.\fct5’.

Warning occurs only for first backing store to bind to this device

Removing Cache Configuration

Nondisruptively unbind cache device from backing store

40

C:\Users\Administrator>dc-unbind C:\MSSQL\tpce_tempdbAbout to tear down binding:

backing store 'C:\MSSQL\tpce_tempdb’ cached by '/dev/fct5’ cache UUID: 52c1edca-0914-40f7-9275-b23aded2a2e8

Do you wish to continue [y/n]? yUnbinding: [====================] (100%) \Successfully unbound: backing store 'C:\MSSQL\tpce_tempdb’ cached by '/dev/fct5'

Note: Unbinding the backing store invalidates cache contents for that backing store.

Viewing Cache Status & Statistics

Cache status and statistics can be displayed with dc-status command

41

C:\Users\Administrator>dc-status –s

fct5: Fusion-io ioDrive2 785GB, SN:1210D0132Device fct5 cache bindings:

Cache device: Status: enabled, Mode: Write-through Backing store: C:\MSSQL\tpce_tempdb Cache instance: dab3fedd-7338-4ea0-b181-9b1b352fb182 VSU 8: dda8d294-7d71-49bf-9b55-1099ed3aa59a In-use: 13.98GBytes Write stats: Hits: requests 0%, sector 0%, Misses: requests 100% Latency: hits 0.0s, misses 2.37247ms, partials 0.0s, total 2.37247ms Read stats: Hits: requests 72%, sectors 93%, Misses: requests 27% Latency: hits 327.666us, misses 50.3810ms, total 14.1122ms

Cache Statistics in Windows Perfmon

Standard performance counters in Windows® Perfmon—reads/sec, sec/read—will not show correct stats

− Cache hits bypass the part of the Windows stack where statistics are gathered

− Standard Perfmon counters show stats only for IOPS that were writes or read cache misses

directCache provides WMI Perfmon counters for monitoring various cache statistics

42

Cache Statistics in Windows Perfmon

directCache provides Perfmon counters for monitoring various cache statistics

43

directCache Cache Invalidation

Data in directCache cache is invalidated in the following cases:

− System reboot

− Binding/unbinding the backing store (dc-bind/dc-unbind)

− Disabling/enabling caching (dc-disable/dc-enable)

− During SDW SnapRestore® operation (only when directCache NetApp SnapRestore Support feature is installed)

44

Collecting Data For Support/Troubleshooting

fio-bugreport generates a detailed report about the state of the device and directCache

− fio-bugreport runs several ioMemory VSL and directCache commands which must be located in your PATH environment variable for commands to complete successfully

45

C:\Users\Administrator> fio-bugreportReport output: fio-bugreport-20120723_153906.cabGenerating bug report. Please wait, this may take a while...---------------------------------------------------Gathering all Windows Event Logs...DONEGathering Fusion-io Windows Event Logs...DONEGathering System Information...