Unstructured Storage Management im Scale-out Petabyte...

Dell Customer Communication - Confidential

Unstructured Storage Management im Scale-out Petabyte-Zeitalter

Matthias Meppen

Advisory SE, Unstructured Data Solutions

07.06.2019, Gut Ullrichshusen

Dell - Internal Use - Confidential

2 of Y


Key Digital Transformation Initiatives

D I G I T A L E X P E R I E N C E S

I N T E R N E T O F T H I N G S

M L / A I B I G

D A T A

D AT A P O W E R S T H E M AL L


3 of Y


What is Data Capital?

D A T A

Human Capital Intellectual Property

Operations Infrastructure

T R A D I T I O N A L A S S E T S

Data is rapidly becoming organizations’ most valuable asset


CHALLENGE:

Transforming Data Into Value

C R E AT I N G B U S I N E S S

I M PA C T F R O M D ATA

K E E P I N G U P W I T H

D ATA G R O W T H

U N L O C K I N G D ATA

T R A P P E D I N S I L O S


D I G I TA L

E X P E R I E N C E S

I N T E R N E T

O F T H I N G S

• Smart sensors

• Connected homes

• Intelligent factories

• Self-service kiosks

• Logistics and supply chain

• Mobile/Web Apps

• E-commerce

• Social Media

• AR/VR

• Sharing economy

Where Data Capital Impacts your Business

B I G D ATA

• Targeted advertising

• Real-time analytics

• Business Intelligence

• Predictive analytics

• Performance monitoring

M L / A I

• Natural language processing

• Robotic automation

• Intelligent security

• Chatbots

• Autonomous vehicles


Transformation of

Your Data Environment


N O W

U N S T R U C T U R E D

E X P O N E N T I A L G R O W T H

D I S T R I B U T E D

R E A L - T I M E

T R A D I T I O N A L

S T R U C T U R E D

L I N E A R G R O W T H

C O N TA I N E D

B AT C H

Evolving Data Landscape


3 data types that have different characteristics

80% of Data is Unstructured

F I L E

O B J E C T

S T R E A M

D A T A C A P I T A L


Unstructured Data Vision

U N I F I E D D A T A L A K E

S I M P L I C I T Y A T S C A L E

E X T R A C T V A L U E F R O M D A T A

F I L E S T R E A M

O B J E C T

© Copyright 2017 Dell Inc. 10


File H I G H P E R F O R M A N C E

S C A L I N G C A PA C I T Y &

P E R F O R M A N C E

F I L E




11

Dell EMC Scale-Out NAS

S i n g l e F i l e S y s t e m &

O n e N a m e s p a c e

G l o b a l A v a i l a b i l i t y

& P r o t e c t i o n

U n m a t c h e d

S t o r a g e E f f i c i e n c y

S i m p l i c i t y &

E a s e o f U s e

L i n e a r

S c a l a b i l i t y

A u t o m a t e d P o l i c y

B a s e d T i e r i n g



Consume Scale-Out NAS on Your Own Terms

T U R N K E Y A P P L I A N C E

S O F T W A R E I S I L O N S D

E D G E

D E D I C A T E D C L O U D

( G C P , A W S , A z u r e )

Cloud


Benefits of Scale-Out NAS for Your Data Capital

E A S E O F M A N A G E M E N T

I N V E S T M E N T P R O T E C T I O N

D R A M A T I C O P E X R E D U C T I O N

U N M A T C H E D R E L I A B I L I T Y

& A V A I L A B I L I T Y

One FS

IDC/Gartner: Isilon is THE Scale-Out NAS Leader







F I L E S T R E A M

O B J E C T



Object C L O U D S C A L E

S T O R A G E

G L O B A L D ATA A C C E S S

C L O U D - N AT I V E /

M O D E R N A P P S

O B J E C T




16

U n l i m i t e d O b j e c t S t o r a g e

E n t e r p r i s e G r a d e

S e c u r i t y a n d C o m p l i a n c e

4 8 % L o w e r T C O

t h a n P u b l i c C l o u d

Dell EMC Object Storage

C o m p r e h e n s i v e

A P I - b a s e d D a t a A c c e s s

N a t i v e M u l t i - t e n a n t

A r c h i t e c t u r e

B u i l t - i n

M e t a d a t a S e a r c h

ECS



Consume ECS on Your Own Terms

T U R N K E Y A P P L I A N C E

S O F T W A R E D E D I C A T E D

C L O U D M U L T I - T E N A N T

C L O U D



Cloud Native Applications

ECS Supports Your Data Driven Workloads

Tiered Archive Sync & Share Cloud Gateway

Cloud Native Apps

(web/mobile) Analytics IoT Cloud Backup

ECS

Traditional Workloads

Consolidate Archive to Lower Costs

Modernize Existing Applications

Accelerate Cloud Native Apps







F I L E S T R E A M

O B J E C T



Stream S T R E A M I N G D ATA

I N G E S T

R E A L - T I M E I N S I G H T S

S T R E A M



Unstructured Storage

Management

Industry Example - ADAS

Data Lifecycle for

ADAS



Disk Load Station

Data Enrichment & Labeling

Test Cases

Test Results

Analysis, Reporting

& Management

Deep Learning Environment

HiL Server Farm Physical devices

HiL/SiL Test Environments

SiL Server Farm

Sensor Sensor Sensor

ECU ECU ECU

Ingest

File-Object

Archive

1

2 3

4

Legend:

1. Ingest Vehicle Data

2. Data Enrichment

3. Data Management

4. Simulation / AI

5. Archive Data

5

ADAS development lifecycle

Test Vehicle Data: Video

Radar

Lidar

GPS

Ultrasonic

Vehicle Data

And More



Single Sensor - SAE LEVEL 2 - (200,000 km)

Storage Requirements:

Scalability

Predictable Performance at Scale

Data Management

EXAMPLE OF DATA CAPACITY REQUIREMENTS

Example: A single Front Looking Radar, operating at 2800 MBit/s (raw)

• 2800 Mbits/s 1260GBytes/hr

• 200,000km, 60km/h ave 3333.3 hours

• 1260 GB/h * 3333.3h = 4.2 PB (raw), 0.9 PB (at ECU)

Multi Sensor - SAE LEVEL 2 vs 3 - (200k vs 1M km)

Sensor Fusion Example:

• Multiple sensors (Lidar, GPS, Ultrasonic, Radar, telemetry, etc)

SAE Level 2

4-10 PB 50-100

PB

SAE Level 3



Potentially hundreds of HIL servers read streams from an Isilon system in parallel

Each stream typically 150-400MB/s (growing with larger resolutions)

Example: 80 HIL servers reading at 200-400 MB/s results in up to 32 GB/s throughput

At same time, new data must also be ingested (multiple GB/s)

Read

Results

32 GB/s Ingest

Single radar sensor

350MB/s

HIL-Server farm Physical Devices

Real time stream

Results

Sensor

ECU

Sensor

ECU

Sensor

ECU

Data performance requirements



Ho

t T

ier

Warm

Tie

r C

ool T

ier

A2000

A2000

H500

H500

H500

F800

TCO optimization: Policy-based automated tiering

Highest performance

Lowest tco

• Tier Between Performance,

Enterprise and Archive Nodes

• Transparent to Users and

Applications

Just Enough Flash



Zero Re-Sim SLA

F800 for DL,

H500 for Simulation Data

No Archiving

Medium Re-Sim SLA

F800 for DL,


Archiving to A2000 Tier

Long Re-Sim SLA

F800 for DL,


Archiving to ECS Object

Ho

t

Tie

r W

arm

Tie

r C

old

Tie

r

ECS

ECS

ADAS SLA Options – 1 day / 7 days / 30 days

Ho

t T

ier

Warm

Tie

r C

ool T

ier

A2000

A2000

H500

H500

H500

F800 F800

Ho

t T

ier

Warm

Tie

r

H500

H500

H500

F800

H500

H500

H500

H500

H500



Use-Case “ADAS Development”







F I L E S T R E A M

O B J E C T



DataFrameworks - ClarityNow

Data

Visibility

FILE OBJECT

Data

Mobility

Data

Access

Data

Search



SITE 1 SITE 2 SITE 3

Cloud

FILE Access OBJECT Access

FILE OBJECT

Active file-object access



CORE Data Centre Hosted Data Lake

CLOUD Expansion

CloudPools - Policy based

automated cloud tiering

SITE 1 SITE 2 SITE 3

Cloud

Archive tiering to cloud

Unstructured Storage Management im Scale-out Petabyte...

Documents

Transcript of Unstructured Storage Management im Scale-out Petabyte...