DDN Product Update from SC13

© 2013 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change.

ddn.com

Store Process Analyze Collaborate Archive Cloud

#BurstForward Invent Discover Compete

1

Product Update

Jeff Denworth VP, Marketing


ddn.com 2

DDN End-to-End HPC Cluster Expertise

Networking Ethernet, InfiniBand,

WAN/Cloud

Schedulers SFX APIs: Prefetch,

User Priority

Cluster Mgr. Dynamic Cluster

Optimization (End-End)

Compute SFA-E, IME

Converged Systems

Applications Ex. 2x-5x 3DSRMME

Gains


ddn.com

Product Portfolio: Three Strong Pillars

Platforms SFA: Block and File,

SSD and Disk, Processors and Storage

Analytics [x]Scalers: Accelerating Time to Insight Across Markets and Industries

Object Storage WOS: Cloud and Data Centers,

Internet, Web and Telcos, Massive Scale and Low Latency

3


ddn.com

Big Data & Cloud Infrastructure DDN 2013 Announced Product Portfolio

4

Analytics Reference Architectures

EXAScaler™ 10Ks of Clients

1TB/s+, HSM Linux HPC Clients

NFS & CIFS

Petascale Lustre* Storage

Enterprise Scale-Out File Storage

GRIDScaler™ ~10K Clients 1TB/s+, HSM

Linux/Windows HPC Clients NFS & CIFS

SFA12Kx 48GB/s/1.4M IOPS 1,680 Drives: 16U-2 Racks Embedded Computing

SFA7700 13GB/s, 400K IOPS 60 Drives in 4U; 396 Drives in 20U

Storage Fusion Architecture™ Core Storage Platforms

SATA SSD

Flexible Drive Configuration SAS

SFX Automated Flash Caching •  Read •  Write*

•  Context Commit •  Instant Commit

WOS® 3.0 32 Trillion Unique Objects

GeoReplicated Cloud Storage 256 Million Objects/Second

Self-Healing Cloud Parallel Boolean Search

Cloud Foundation

Big Data Platform Management

DirectMon

Cloud Tiering

* Future Release

WOS7000 60 Drives in 4U

Self-Contained Servers


ddn.com

Big Data & Cloud Infrastructure DDN 2013 Announced Product Portfolio

5

Analytics Reference Architectures

EXAScaler™ 10Ks of Clients

1TB/s+, HSM Linux HPC Clients

NFS & CIFS

Petascale Lustre* Storage

Enterprise Scale-Out File Storage

GRIDScaler™ ~10K Clients 1TB/s+, HSM

Linux/Windows HPC Clients NFS & CIFS

SFA12Kx 48GB/s/1.4M IOPS 1,680 Drives: 16U-2 Racks Embedded Computing

SFA7700 13GB/s, 600K IOPS 60 Drives in 4U; 396 Drives in 20U

Storage Fusion Architecture™ Core Storage Platforms

SATA SSD

Flexible Drive Configuration SAS

SFX Automated Flash Caching •  Read •  Write*

•  Context Commit •  Instant Commit

WOS® 3.0 32 Trillion Unique Objects

GeoReplicated Cloud Storage 256 Million Objects/Second

Self-Healing Cloud Parallel Boolean Search

Cloud Foundation

Big Data Platform Management

DirectMon

Cloud Tiering

* Future Release

WOS7000 60 Drives in 4U

Self-Contained Servers

Lots of Amazing Products More Than We Can Cover Today... Focus For This Talk: •  DDN’s HPC Engineering Agenda •  SC’13 New Product Update


ddn.com 6

Our HPC Focus

Infinite Scale Research

Collaboration & Data Ubiquity

Simple Infrastructure

Price /Performance


ddn.com 7

Razor-Focused On The Cost of Computing

Price /Performance

Fully-Parallel, State Machine I/O Best HW Efficiency Least Electronics / perf. unit

HPC’s Most Aggressive Caching Strategy •  HW: RAM, Flash •  SW: APIs, Buffers Save on HDDs, Racks, Power Increase Cluster Efficiency

Compute/Store Convergence •  Eliminate I/O Server Layer •  Buffers can save $MMs

Data Center Density & Storage Capacity •  Best Slot per RU •  Least Electronics / perf •  Most Disks / system •  WOS = 99% platter util.


ddn.com 8

Breaking Down Barriers To Data Access.

Aggressively Enabling Multi-Protocol Platforms Single Namespace, Ubiquitous Data •  Parallel File Systems •  Cross-Protocol NAS •  Even Object Storage DDN is actively engineering multi-protocol namespace management

Protocol Barriers Geographic Barriers

Cloud Storage Technology Can Enable Rapid Insights •  Massive Namespace •  Easy OID Management •  Robust Sync Tools •  Advanced, WAN Data Protection Add Parallel File Systems, and Presto!

Research Collaboration

& Data Ubiquity

New: DDN GRIDScaler New: DDN WOS Bridge


ddn.com 9

Infinite Scalability Delivers True Power To Your Apps

Engineering Around POSIX Locking •  Totally Distributed Algorithms •  Buffering to Relieve I/O Pressure

Namespace Level Storage Level

Real-Time Data Delivery (HDD) •  Guaranteed Response SLAs •  Eliminate Rebuild Latency Non-Deterministic Data Placement & Retrieval •  Eliminate File Layout Hostpots (Buffer) •  Load Aware Write & Read (WOS, Buffer)

Infinite Scale


ddn.com 10

Sophisticated I/O Technology, Management Simplicity.

Simple Infrastructure

One Tool; Any Platform SFA, GridScaler, ExaScaler, Clusters, Hadoop, WOS

Self-Notifying System Management Now supporting Call Home!

Compute/Store Convergence DDN SFA Appliances eliminate I/O servers & networking to reduce systems by <90%

Self-Healing Storage •  SFA: Drive Power Cycling •  WOS: Data Protection

Policies Always-Conform


ddn.com 11

Proven Blueprints; Optimized for TCO Scale and Speed

And more...


ddn.com 12

One Integrated HPC Storage Stack from The HPC Leader

WOS™ Bridge

Cloud Connector and File System Data Replicator

SFA™ Storage Appliances

ReACT™ and SFX™ Storage Caching

DDN Scale-Out Parallel File Systems

Multi-Protocol Access Infinite Memory Engine™

Burst Buffer System

Dire

ctM

on™

Parallel File System Clients

NAS-Attached Clients

DDN Delivers the Most Powerful & Comprehensive End to End Stack for HPC


ddn.com 13

The World HPC Champion is Now Even Faster!

Introducing The SFA12KX Family

§  World-Leading Parallel File I/O Foundation §  Best-In-Class IOPS & B/W for All Workloads §  Multi-Level Write and Read Cache, Cache APIs §  Deep Capacity, Leading Data Center Density §  Server-Virtualized To Embed HPC File Systems

SFA12KX 48GB/s

SFA12KE 23GB/s

2 New Models:


ddn.com 14

SFA7700: World’s Fastest Scale-Out Hybrid Platform DDN’s Departmental Storage Powerhouse

The Speed of Flash, The Cost of Disk. The Power of SFA-OS, All in Only 4U Fully-Featured Like the SFA12KX §  SFX Flash Cache and API-Enabled Caching §  Next Gen DirectMon Appliance Management §  World Leading Hybrid Price Performance §  13 GB/s, 400K Cache IOPS, 900TB in 12U SFA7700


ddn.com 15

DDN Revolutionary Caching Boosts HPC Application Performance

70% of I/Os are <512KB, 50% are <16KB 58% write I/Os vs. 42% read I/Os

Caching Systems Are HPC’s Shock Absorbers •  DeBlends Unaligned Writes •  Preserve Small I/O Performance •  Smoothes Out Mixed I/O Inconsistency •  Significantly Improves Storage Utilization

Source: ORNL

Write Distribution for Multi-Disciplinary HPC Cluster Sampling from ORNL Testing


ddn.com

SFX Improves Application Performance by Up to 700%

DRAM Cache

HDD Tier SFX Write*

Part

ial W

rites

to R

AM

Alig

ned

Writ

es To

Fla

sh

ReACT

Cache Flush

Cache Warm

SFX Read Instant Commit

Context Commit API In-Band - Out Of Band

Usage & Priority-Based Caching

* 2014

16


ddn.com

DDN Application-Defined Caching Eliminates Cluster Idle Time In-Band File System Hint Awareness; Out-of-Band Middleware & DB Structure Caching

I/O CPU

Time

I/O CPU

SFX Prefetch

SFX Prefetch

I/O Idle I/O Idle File System Client

Operating System

Application Scheduler

Time

Out

of B

and

Ethe

rnet

Load

Load

Out-of-Band API Control Enables Dramatic Efficiency Gains

SFX is equally valuable in telling the system what not to cache

API

File System ‘Hint Sniffer’ [Technical Term J]

17


ddn.com 18

DDN Boosts File System Speed, Access and Reliability

Our Value-Add Approach To Accelerate I/O, Minimize TCO & Facilitate Collaboration.

§  Open-Platform File System Agenda §  Data Ubiquity: NAS, Parallel IO, Cloud §  Caching Tools To Accelerate I/O, APIs §  Collaborate and Archive Via The Cloud §  Easy To Manage Any Platform, Any Scale

GRIDScaler™

EXAScaler™


ddn.com 19

WOS 3.0: The World’s Most Scalable Web Storage Simple, Hyper-Efficient Object Storage from The Performance Leader

99.99999%+ Best-In-Class Cloud Availability

32 Trillion Object Scalability

10TB/s throughput

256M objects/sec

Up To 64 Remote Sites

API, REST, NAS Access


ddn.com 20

WOS Delivers True Cross-Campus and Global Collaboration

Site 3

Site 4

Compute Grid

WOS™ Storage Cloud

Site 1

Site 2

Compute Grid

Compute Grid

Compute Grid

•  Eliminate Research Data Silos •  ‘Push Model’ Data Distribution •  Synchronize Independent File

Systems for Simple Replication •  Access Data From A Variety of

Protocols


ddn.com

DirectMon Next Generation Monitoring Tool

•  New Web Interface •  Deep Statistical Monitoring •  SFA Configuration Tools •  ‘Scaler Dashboard & Config •  Log & Config Management •  Call Home Enabled in Q4 •  Scale-Out Cluster Manager

21


ddn.com 22

DDN Delivers the Most Powerful & Comprehensive End to End Stack for HPC

One Integrated HPC Storage Stack from The HPC Leader

WOS™ Bridge

Cloud Connector and File System Data Replicator

SFA™ Storage Appliances

ReACT™ and SFX™ Storage Caching

DDN Scale-Out Parallel File Systems

Multi-Protocol Access Infinite Memory Engine™

Burst Buffer System

Dire

ctM

on™

Parallel File System Clients

NAS-Attached Clients


ddn.com



Exascale I/O Platform Technology Introduction


ddn.com 24

Evolution of Data-Intensive Computing

2010

12 Threads per Node

CPU Average Top500 Cluster Concurrency

13,000 Cores

2018 (est.)

O(1,000) Threads per Node 57,772,000 Cores 4,400x Increase


ddn.com 25

File Systems Will Not Survive Hyper-Concurrency

0 10 20 30 40 50 60 70 80 90

100

1 10 100 1000 10000 100000

Conc

urre

ncy

Deg

ree

(Per

cent

)

File System Degradation vs. Concurrency

Wall-Clock Degradation

<Exascale> <~2015> <Today>

Source: DDN Testing; Exascale Research

Million and Billion Core Computers Need Smarter Solutions To Scale Applications and Meet Uptime Requirements


ddn.com 26

NVRAM Presents New Architectural Opportunity

1

100

10000

1000000

100000000

1E+10

2000 2010 2018 HDDs (IOPS) Flash CPU Operations

And Performance Gains Outpace HDD

Widening 106 to 108

GHz

IOPS

IOPS

NVRAM Is A Viable HPC Tier, Today

$/MB/s: $0.29 (91% less) $/GB: $1 (1000% more) W/GB/s: 0.003 (98% less)

$/MB/s: $3.19 $/GB: $0.1 W/GB/s: 0.13


ddn.com



27


ddn.com

Introducing a Revolutionary Exascale/Everyscale Architecture

•  Dramatically Reduces HPC Computing Cost •  Enhances Performance & Power Usage by Orders of Magnitude •  Reduces Network & Server Amount & Complexity by Orders of Magnitude •  Patented Algorithms Deliver Unlimited Scale •  Liberates Applications From File System Bottlenecks

Leveraging 15 Years of HPC Expertise, Engineering & Deployment Leadership

28


ddn.com

Next-Generation Distributed Memory System from the Worldwide HPC Leader

•  Speed Up Cluster I/O 1000x •  Divide Number of Disk Drives by a Factor of 10 •  Divide Rack Space by a Factor of 10 •  Divide Power Needs by a Factor of 10 •  Break Through POSIX Scaling Barriers •  Achieve Unlimited File System Scalability

29


ddn.com 30

Infinite Memory Engine Delivers Unparalleled Scale

0

50

100

150

200

0 1 2 3 4 5 6 7 8 9

Frag

s/Se

c (M

illio

n)

Number of IME Nodes

IME Metadata Scaling (24 Concurrent Threads)

IM 'Max' Trinity-Pessimistic-Req Proven IME Scaling

200 PFLOPS Bid Requirement

Linear Scalability!


ddn.com 31

The Infinite Memory Advantage

Limitless Scalability Patented DDN Algorithms

Scale-Out Data Protection Distributed Erasure Coding

Integrated With File Systems Designed to Accelerate Lustre*, GPFS No Code Modification Needed

Fully POSIX & HPC Compatible No Application Modifications

Non-Deterministic System Write Anywhere, No Layout Needed

Writes: Fast. Reads: They’re Fast Too. No other system offers both at scale.


ddn.com 32

IME Completes DDN’s HPC-Leading Caching Strategy The Ultimate I/O Deblender

SSD 50μs

HDD 17ms

RAM 7μs

SFX Flash Tiering

API Control

SFA Storage Fusion Architecture


ddn.com 33

IME HW Savings Grow As Moore’s Law Advances

Average HPC Cluster Example (2018)

Machine Size/Checkpoint 100M Threads, 2PB Checkpoint in 5m

Disk Only Node Local IME + SFA

Peak Performance 7 TB/s 8 TB/s

Spinning Disks 33,400 3,240

Racks 100 12

Storage Acquisition Cost $33M $7M 62% 80% 0%

20%

40%

60%

80%

100%

2016 2018

IME Savings Grow Greater over Time

The Growing HDD Gap


ddn.com



34

Live Demo Running In Denver!


ddn.com 35

Worldwide Debut!! Running Live In DDN’s SC’13 Booth.

Node Local Buffer Storage Topology

Hybrid Compute & Storage 14 x Commodity Compute Nodes Each Node with 7 Low-Cost, Node-Local SSDs

High-Speed Persistence Layer File System Replay to SFA7700 w/ Lustre* Near-Perfect Disk Utilization; Fully-Ordered I/O


ddn.com 36

Demo Comparative Testing: Shared Writes 14 IME Clients (Dell nodes); 98 SSDs

DISK LEVEL TESTING DDN GRIDScaler™ (per SSD)

IME (per SSD)

62.5 Concurrent Write Requests 438 MB/s 500 MB/s

125,000 Concurrent Write Requests 170 KB/s 500 MB/s

CLUSTER LEVEL TESTING DDN GRIDScaler™ IME( overall)

6,225 Concurrent Write Requests 43 GB/s 49 GB/s

12,250,000 Concurrent Write Requests 17 MB/s 49 GB/s

SSDs behind a PFS don’t help IME is at line rate to scale with SSD rates

2,669x More Metadata Scalability Simulates System 1/4th the size of The Average Top500 in 2018.

Linear Cluster Scaling

Avg. 2018 Top500 Cluster Concurrency 57,772,000 Cores (est)


ddn.com

From Exascale to Everyscale, DDN Is THE HPC Leader

The Worldwide HPC Storage Leader ▶  Liberating Applications From

Infrastructure & File System Scaling Limits ▶  The Most Advanced HPC Stack To

Minimize The Cost of Computing ▶  Unleashing Global Collaboration ▶  Simplifying Scale-Out for Everyscale

37


ddn.com 38

See You On The Floor...

•  We Have Terrific Speakers Lined Up Today! •  Over 90 People From DDN At SC’13 •  CTOs & Engineers Are Here In Force

•  Discussing Anything/Everything HPC •  ExaScale to EveryScale

•  Geek Out On Our $M Demos •  WOS, SFA, IME, Lustre*, Analytics, Etc.

•  Plan The Future of HPC In Our Whisper Suites

See You @ Booth 1307

DDN Product Update from SC13

Technology

Transcript of DDN Product Update from SC13