Emerging Technologies for HPC Storage › uploads › 7 › 2 › 9 › 8 › 72981523 › 06... ·...

Emerging Technologies

for HPC Storage

Dr. Wolfgang Mertz

CTO EMEA Unstructured Data Solutions

June 2018

© Copyright 2018 Dell EMC2

The very definition of HPC is expanding

Blazing Fast Speed Accessibility and flexibility

Traditional HighPerformance Computing

Computationally-intensive modeling and simulation

• Computer-aided engineering

• Weather forecasting

• Oil exploration

High Performance Data Analytics

Complex or time-critical ‘big data’ analytics workloads

• Genomics

• Financial analytics

• Business intelligence

Artificial Intelligence

Machine and deep learning

applications

• Fraud / anomaly detection

• Predictive maintenance

• Personalized medicine


HPC vs HPDA from a data perspective

Start EndTime

HPC

HPDA


µs ms s hour day month yr+

“Fast Data”

“Big Data”

year

Value of

Data ($)

Time

Value of data over time…


HPC Storage Challenges

Performance

Persistence

Performance

High Availability,

Backup, Data

Protection

Data Sharing and

Accessibility

Management &

IntegrationCritical Challenge

bare metal, parallel access Not the critical featureUptime important, but no

special backup

requirements

Important, but not the

critical feature

Scalable performance,

tunable for workload

Able to fulfill compliance

requirements, protect

important data

Pre and post

processing, analytics,

desktop access

Management

functionality and

support; connections

with other tools

Traditional ‘/scratch’

Traditional ‘/home’


Pe

rfo

rma

nc

e

Capacity

HPC NFS Storage Elastic Cloud Storage

(ECS)

Virtustream

HPC Fast Storage(Lustre, BeeGFS, GPFS)

NVMe-oF

Job

Scheduler

Dell EMC

&

3rd Party

Isilon

Scratch Project Archive


NVMe Usage Scenarios: Local Dedicated Devices

Up to 24 U.2

NVMe (R740XD)

Targets: Up to 4 Intel AIC

NVMe P3700 2TB

Each host has

one or more

dedicated

NVMe devices

Up to 24 U.2

NVMe (R740XD)


NVMe P3700 2TB

Up to 24 U.2

NVMe (R740XD)


NVMe P3700 2TB

Up to 24 U.2

NVMe (R740XD)


NVMe P3700 2TB


NVMe Usage Scenarios: Sharing NVMe over Fabrics

NVMeF Host

Client

NVMeF Host

Client100 Gb/s

Each host can mount one or

more NVMe Targets.

Targets: Up to 24 U.2

NVMe (R740XD)


NVMe P3700 2TB

Targets U.2 NVMe


NVMe P3700 2TB

Even hosts with no space or

support for NVMe can use

NVMeF devices.


NVMe-oF Test System 1

Mellanox EDR

ConnectX-5

R730 NVMeF Targets

Server

E5-2690 v4 @ 2.60 GHz.

256 GiB DDR4 2166 MHz

R730 NVMeF

Host Client 1

R730 NVMeF

Host Client 2100 Gb/s

Targets: 4 Intel AIC

NVMe P3700 2TB

2x Xeon E5-2690 v3 @ 2.60 GHz


Each host can mount one or more

Targets.

RHEL 7.4 x86_64 (GA level), kernel version 3.10.0-693

Native Drivers


NVMe-oF Test System 2

OmniPath

R730 NVMeF

Host Client 1

R730 NVMeF

Host Client 2100 Gb/s

Targets: 4 Intel AIC

NVMe P3700 2TB

R730 NVMeF Targets

Server

E5-2690 v4 @ 2.60 GHz.


2x Xeon E5-2690 v3 @ 2.60 GHz


Each host can mount one or more

Targets.

RHEL 7.4 x86_64 (GA level), kernel version 3.10.0-693

Native Drivers


Configuration Details of NVMe-oF

• 3 Dell PowerEdge R730, one used as (target) server connected to two clients (hosts).

– Clients Dual Intel Xeon E5-2690 v3 @ 2.60 GHz.

– Server Dual Intel Xeon E5-2690 v4 @ 2.60 GHz.

– 256 GiB of DDR4 @ 2133 MHz.

• Omni-Path adapters installed on slot 4 (PCIe x16) connected to a switch.

• RHEL 7.4 x86_64 (GA level), kernel version 3.10.0-693

• 4 Intel P3700 2TB AIC adapters on the server slots 1, 2, 3 & 5 (all PCIe x8)

• FIO 2.99 compiled on each machine with libaio support

• DirectIO and no buffered IO was used to prevent RAM cache

• Ramp up time was one hour and test time was limited to 300 seconds for each data point

• Each write test with a different block size was followed by a consistent read test


OmniPath (100 Gb/s) BW Average

Bandwidth Baselineib_write_bw -F -R -a 172.20.1.1

0

2000

4000

6000

8000

10000

12000

2 4 8 16 32 64 128 256 512 1K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M 4M 8M

MB

/sec

Package Size

Infiniband EDR (100 Gb/s) BW Average


Going to Next Level with Hardware – ME4• RAID Array: 2u12, 2u24, 5u84

• Expansion: 2u12,2u24,5u84 (DAE)

• Backend Interface:12G SAS

• FE Interface:

• 16 FC 4 ports per controller

• 10G iSCSI 4 ports per controller (SFP+ or BaseT)

• 12G SAS 4 ports per controller

• Reads IOPS: 320K – 4x over MD3

• Seq. Reads: 7000 MB/s

• Seq. Writes: 5500 MB/s – 2.6x over MD3

• Total system drive count: 336 -1.75x over MD3

• Raw Capacity: 4PB

• Single or Dual Controller

ME4012: 12-drive RBOD



ME412: 12-drive Expansion



Note: ME Expansion Units (DAE) cannot be connected

to a server directly (not a server-attached JBOD)


Parallel File System – Lustre w/ ME4

Dell PowerVault

ME4024

Dell PowerVault

ME4024

(optional for DNE)

Dell PowerVault

ME4084


Dell EMC Isilon Scale-out NASHigh Performance to Archive • Few TBs to 100 PB in a single file system

– Up to 1.5 TB/sec Aggregate Read

• Policy-based Automatic Tiering

– Flash, SAS, Sata

• Native Multi Protocol Access

– NFS, CIFS, HDFS, Swift

• Enterprise Features for Data Management, Long Term

Archive and Compliance

F-Series

Tier 1

A-Series

Tier 3

H-Series

Tier 2

Re

du

ced

cost/T

B

Files

CloudPools


ECS Scale-Out Object Store

Modern archive Universal archive for existing primary storage. Replaces tape. No changesto applications or operations. Archive always online of analytics workflows

Cloud nativeEnable new healthcare business operations. Cloud economics andease of use on-premise. Lower TCO compared to public cloud providers

ScalabilityDeployable in clusters for petabyte and exabyte scalability

Data protectionProvides geo-distributed data protection with no single point of failure. Globally accessible. One namespace. Multi-tenant architecture

Accelerate cloud native applicationsFuture healthcare IoT applications on private infrastructure

Operational flexibilityMulti-protocol support for legacy & modern applications


Emerging Technologies for Persistent Storage

• Higher Scale

– 100s PB for File

– Exabyte for Object

• High Performance Object

• Removing Protocol Overheads

• Gen 7

Emerging Technologies for HPC Storage › uploads › 7 › 2 › 9 › 8 › 72981523 › 06... ·...

Documents

Transcript of Emerging Technologies for HPC Storage › uploads › 7 › 2 › 9 › 8 › 72981523 › 06... ·...