Storage Industry Overview for the Disney Architecture · PDF fileStorage Industry Overview for...

© 2008 IBM Corporation

The Challenge of Scale

Storage Industry Overview for the Disney Architecture Council

Clodoaldo BarreraChief of Technical Strategy

IBM Systems Storage

2


� 50+ years in storage business

� IBM System Storage – > $5 billion business

� ~5,000 people, 170 countries, 1,000+ BPs

� 15 development labs worldwide

� $500 million + in R&D annually

� Broad industry partnerships

� Storage innovation leadership► 100s of patents in 2006

� Full portfolio of systems and solutions offerings► Information lifecycle management, business continui ty,

infrastructure simplification

► Disk, tape, SAN/NAS, software, services, financing

IBM System Storage

3


Summary� We see new data types, new business uses of

information, and new rules for the preservation and security of information driving rapid growth in storage capacity.

� These uses will require new approaches for scale, performance, protection, and lower cost.

� Progress in basic recording technology, computing, and networking will not eliminate all of the issues

� Investments in new infrastructure and processes will be needed to prepare for the coming uses for digital information

4


External disk shipments & price (History & Forecast)

173 239 306 462784

1306

2409

3223

5026

7784

11977$126.00

$72.20

$43.90

$31.20

$19.55$13.34

$9.10$6.13 $4.14 $2.80 $1.890

2000

4000

6000

8000

10000

12000

14000

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20100

20

40

60

80

100

120

140

PB

$/GB

External storage capacity growth trend

Exabyes shipped & WW disk revenue

0

5

10

15

20

25

30

35

2005 2006 2007 2008 2009 2010

WW

Rev

enue

$B

0

2

4

6

8

10

12

14

EB

shi

pped

Source: IDC, 2007

5


Storage Workloads - Classic and New Use cases

Transaction Processing

Business Intelligence

File Serving

Backup & Data Protection

Disaster recovery

HPC

Data WarehousingStorage Security

Automated Management

Video Serving

Medical Imaging

Searchable Archives

Web 2.0Grid File Serving

Digital Video Surveillance

6


Storage Workloads: Profile for Financial Markets

BackupWeb/Grid

Low LatencyOLTP

AnalyticsEnterprise Computing

Storage MgtData MigrationCommon Svcs

Archive

FC/FICON SAN

IP SANNAS

FC/FICON SAN

FC/FICON SANNAS

IP SAN & NASFC

IP SANNAS

High IOPLow latencyModerate-large capacityHi performance diskLarge cachesSpecialty DR, backup- Very low RTO/RPO- Synch. Remote mirror,3rd site

Highest data integrity Non-Stop Operations

Mixed workloads- small record and streaming

Large and growing capacityNeed low cost operationsNearly non-stop operationsDR and backup- Low RTO/RPO- Synch. Remote mirrorHigh data integrity RO/BO Support

Storage for ClustersLoad (Sequential), QueryHigh throughputStream analytics

DR - Duplicate facilities- Common data

feed

Sequential WorkloadIntegration with appsDisk and Tape hierarchy- Lowest cost through tiersDeduplicationData ExpirationMedia Management

Seq. Fileand Objects

Indexes for searchDeduplication

Disk and Tape Hierarchy

Low cost, large capacityDuplicated dataModest performance- Low performance transactionsDR

- Duplicate facilities and data

Low management costEase of deployment

7


External Disk Industry by Segment

9441

158212651366

2325

8865

1741

1820

1530

2799

8398

1917

2359

1741

3114

7535

2110

3147

1941

3394

6854

2323

3656

2161

3674

5818

2558

4388

2401

3831

4589

2816

5265

2664

3990

$0

$5,000

$10,000

$15,000

$20,000

$25,000

$M

2006 2007 2008 2009 2010 2011 2012

Year

External Disk Spending by Segment($>50K)

Traditional Apps - Addressable (>$50K) NAS Addressable Market (>$50K)

Archive - Addressable (>$50K) Web 2.0 - Addressable (>$50K)

Embedded Apps-Addressable (>$50K)

CAGR2007-2012

•Embedded 10.0%

•Web 2.0 11.7%

•Archive 20.0%

•NAS 10.1%

•Traditional -12.3%

2.4%$15,340$15,164$14,995$14,734$14,416$13,957$13,654External Disk $50K + (GMV2H07)

CAGR 2006-10

2012201120102009200820072006External Disk $50K+ Opp., MF and Open

Source IBM 2007

8


Storage Infrastructure Layers

Device Access, Pathing

Backup/HSM

File SystemsContent Management

Storage Virtualization

Storage Devices

DBMS

Applications


Storage M

anagement

Security M

anagement

Archive Replicate

System

s Managem

ent

9


Styles of Storage Infrastructure

Application Layer

Business Policy

DBs / FS

Block Data Access

SCSI/SAN

Block Storage

Managem

ent

Storage Network

Transaction Paradigm

Application Layer

Business Policy

DBs / FS

File Data Access

Local/Cluster/Network

File Server

Managem

ent

Block Storage

IP Storage Network

File/Object Paradigm

Managem

ent

Application Layer

Business Policy

Cluster FS

File Data Access

Local/Cluster/Network

File Server

Managem

ent

Block Storage

HPC Paradigm

Mgt

(Specialty Data Manger)

Cluster Storage Network

MG

T

10


Storage Devices

11


The Device Dilemma – 1Cost takedown is slowing

Full History Disk Areal Density Trend

0.000001

0.00001

0.0001

0.001

0.01

0.1

1

10

100

1000

1950 1960 1970 1980 1990 2000 2010

Year of Production

Gb/

sq. i

n.

60-100% CAGR

25% CAGR

60-100% CAGR

Disk Areal Density Trend 2000-2010

0.1

110

1001000

10000

1998 2000 2002 2004 2006 2008 2010

Year of Production

Are

al D

ensi

ty

Gb/

sq.in

.

100% CAGR

25-35% CAGR

Historic trend

� From 1957 until today CAGR of Areal Density as averaged 30-35%/yr

� Between ~1990 and ~2004 it averaged 60-100%/yr

Current Decade

� Around 2004 the CAGR dropped to 25-35%/yr

� Disk vendors indicate that it will continue at 25-35% CAGR for the rest of the decade

� This will have a significant impact on the price and performance of disk drives through the end of the decade

Action: Return to capacity management discipline, ILM, Dedup and CompressionAction: Return to capacity management discipline, ILM, Dedup and Compression

12


The Device Dilemma – 2 Performance / Capacity losing ground

1990 1995 2000 2005 2010

Year

0.1

1

10

100

1000S

usta

ined

HD

D I/

O R

ates

per

GB

yte

Desktop and Server Drive Performance

<=7200 RPM

10K RPM

15K RPM

Desktop5400 and 7200 RPM

Action: Need SSD’s and QOS on LUNs for high IOP/GB workloadsAction: Need SSD’s and QOS on LUNs for high IOP/GB workloads

13


The Device Dilemma -3 Device Reliability Improves Slowly

� Drive reliability (MTBF and hard error rates) not keeping up with capacity growth.

► Implies need for multi-parity RAID designs

1996 1997 1998 1999 2000 2001

Year of Drive Introduction

0

20

40

60

80

100

in k

hou

rsM

ean

Tim

e to

Fai

lure

per

GB

High/GB Low/GB

1985 1990 1995 2000 2005

FCS Date

0

200

400

600

800

1000

1200

1400

Spe

cifie

d M

TB

F, k

hrs.

IBMSeagateDECMaxtorHitachi

Ed GrochowskiAlmaden Research Center

HDD MTBF Manufacturer Specifications

Action: Need to address upper layer system resiliency

Action: Need to address upper layer system resiliency

14


The Device Dilemma 4 - PowerComponents of Data Center

Power ConsumptionStorage Power Consumption/GB

Data Center Storage UsageExternal PB Shipped

Data Center Storage Power Growth

Cool i ng

S e r v e r &S t or a ge

Conv e r si on

Ne t wor k

Li ght

0

5000

10000

15000

20000

2005 2006 2007 2008 2009 2010

00.20.40.60.8

11.21.41.6

2005 2006 2007 2008 2009 2010

0

0.2

0.4

0.6

0.8

1

2005 2006 2007 2008 2009 2010

Source: IBM Source: IBM

Source: IBMSource: IDC

Action: Virtualization, Storage Tiering, Data migration, Tape, and Solid State (someday)Action: Virtualization, Storage Tiering, Data migration, Tape, and Solid State (someday)

15


Device Directions - Summary

� Deeper RAID protection

� SSDs will be used in high IOP applications

� Multiple device types►SSD, Fast disk, high capacity disk, Tape

� Increased focus on management:►Capacity Utilization

►Power utilization

16


Storage Systems

17


Approaches to very large storage scale

� What Hurts► Manual processes

► Heterogeneity● In architecture, equipment● workloads, and process

► Non-standard technology

► Dependence on fault tolerance, single instance resources

► Disaster recovery as a special operation

� What Helps► Automation

► Homogeneity

► Regular clustering with geographic extensions● Failures expected

► Standards by layer

► Replicated resources

► Disaster tolerance as an artifact of geographic clustering

► Storage Tiering

18


Key Technologies – Directions in Storage

� Storage Virtualization - heterogeneous

� Array Clustering – homogeneous

� Solid State Storage

� Converged Networking

� Scalable File Systems

� Active Archive► ILM

► De-duplication, Compression

► Indexing, search

► Tiered Storage – Including Tape

� Management - The Ensemble Model

19


External Disk Arrays – Classic View

Function

Cap

acity / PR

ICE

/ PE

RF

OR

MA

NC

E

Entry RAID/JBOD•FC/SAS/iSCSI attach•Rack packaging

• Integrated disk & controller

• 2U, 12 drive enclosures • Single/dual controller• No/limited expansion • 12-48 drives

•SAS/SATA drives•Some failure modes

Modular Midrange•FC/iSCSI attach

• 4 interfaces•Rack Packaging

• Integrated disk & controller• Dual controller• Card sized controller –

integrated electronics• Limited expansion – 40-60

drives•FC drives, SAS/SATA drives•Limited data copy functions•Applications

• SMB data center• Distributed operations

Scalable Midrange•FC attach (IB a future option)

• 8+ interfaces•Rack packaging

• Controller package• Board sized controllers• Separate drive packages• Capacity scaling through

multiple drive boxes•FC drives, SAS/SATA drives

• Up to 200 drives•Continuous availability•Data copy functions•Applications

• Enterprise tier 2• Eng/Scientific• Mid-large clusters

High End Enterprise•FC/FICON attach 64+ interfaces•Integrated disk & controller•Cabinet packaging•Expansion Racks•Up to 1000 FC Drives•Continuous Availability•Custom controller designs

• MP Controllers• End-end data checking• High RAS design

•Data copy functions• Snap Copy• Synch and Asynch copy

•Applications• DB and file applications• ZOS, ECKD• Mission critical applications• Enterprise Storage

consolidation

20


Virtualization as a Technology of Scale

21


Hypervisor (Type 1)

Hypervisor (Type 2)

Host OS Storage Virtualization Network Virtualization

Intel, Power, Z

System Lifecycle Management (Virtual & Physical) - ImageCreate Version Deploy Discover Monitor Update (Patch) Audit /Compliance/Secure License/Usage Destroy

Basic Control (VM)CreateDeleteMove/Migrate

CloneSuspendResume

AggregateSegregateSnapshot

Business System Management (Virtual and Physical System) :

Business Continuity QOS (& Capacity Management) System Consolidation Resource Pool (Farm) Mgmt.

Pratik GuptaCTO Office

Process Management:Change Mgmt Configuration Mgmt. Release Management Problem Mgmt Incident Mgmt Availability Management Capacity mgmt

Physical resources Virtual Resources & Aggregations

Platform Management (IBM Systems Director)

Firmware update Hardware monitoring Virt Mgr

Standards Based Interface

Service M

anagement

System Virtualization – Including Storage

22


Storage DevicesPool 1

Virtual Storage – Scale Out Model


Archive

Replicate

Physical S

torageM

anagement

Virtual S

torage Managem

ent

Backup/HSM

Storage DevicesPool 2

Storage DevicesPool N


Storage Provisioning

Archive Replicate Backup/HSM

File Systems

Content Management

DBMS

Applications


Storage Provisioning

Archive Replicate Backup/HSM

File Systems

Content Management

DBMS

Applications

VirtualStorageTier 1



Virtual S

ystems M

anagement

Physical S

ystems M

anagement

Data S

ecurity Managem

ent

23



Customer Challenge Virtualization Benefits

Application Availability

� Reconfiguration

� Migrations

Non-Disruptive Operations

�Physical storage changes are not visibleto application servers

Complexity & Personnel Productivity

� Most customer have storage from multiple vendors

Logical to Physical Mapping

�Allows use of heterogeneous storage to beused regardless of storage provider

�Management of storage can be performedin a common way from a single point ofmanagement

Storage Utilization

� Capacity not efficiently used

� Difficult to manage and monitor utilization

Virtual Storage Pools

�Capacity can be allocated from multiple storagearrays

�Management of capacity utilization can be donefrom a central point

�Pools of different capabilities can be created

24


SANVolume Controller

SAN Volume Controller Supported Environments

SAN with 4Gbps fabric

HPMA, EMAMSA, EVA

XP

HitachiLightningThunder

TagmaStoreAMS, WMS

EMCCLARiiONSymmetrix

MicrosoftWindows 2008

MSCS

MPIO, VSS, GDS

IBM AIXHACMP /XD

GPFS / VIO

SunSolarisVCS/SUN

clustering

HP-UX 11i V3Tru64

OpenVMSServiceGuard with SDD

Linux(Intel/Power/zLinux)

RHEL/SUSE

RHEL 5 ia32, x64

RHEL 3 Power

SLES 9 ia64

IBMBladeCenter

Win/Linux/VMWare/AIX

OPM/FCS/IBS

SAN

SANVolume Controller

Continuous Copy

Metro Mirror

Global Mirror

VMware

Point-in-time CopyFull volume, Copy on write

256 targets, Incremental, CascadedSpace-Efficient

NovellNetWareClustering

SunStorageTek

IBMDSDS3000

DS4000

DS6000

DS8000

IBMESS,

FAStT

1024Hosts

iSCSI to hostsVia Cisco IPS

IBMN series

NetAppFAS

SGI IRIX

IBM N series GatewayNetApp V-Series

BullStoreWay

FujitsuEternus

NECiStorage

For the most current, and more detailed, information please visit ibm.com /storage/svc and click on “Interoperability

Space-Efficient Virtual DisksUp to 8192 Virtual Disks

Virtual Disk Mirroring

AppleMac OS

PillarAxiom300, 500

25


Block Virtualization Competitive Landscape Assessme ntCurrent Products vs. Virtual Storage Infrastructure Requirements

EmergingSeems to be difficultYesHitachi USPMature Implementations

potential to evolve

Potential, but architecture makes sharing some

services more challenging

Yes

no appliance between host & disk

Manual, volume-level migration; potential to evolve

No appliance in between create MPIO driver

challenges

Yes

Inherently the most scalable

EMC Invista, StoreAge, InServ/Incipient*

Split-Path or Switch Based

Lefthand SAN/iQ, CompellentXIV, Pillar Data

IBM SVC, FalconStor IPStor, DataCore SANSymphony

HDS – TagmaStor (also OEM’d by HP & Sun)

Competitors / Products

potential to evolve

Shared replication, some limits (currently)

3rd party storage attached to slower bus

Manual, volume-level migration; potential to evolve

Slower bus & some function restriction for 3 rd party

storage

Within single storage array

Tied to single storage array

3rd Party Attach to Traditional Array

potential to evolve

replication services shared; great potential for evolution

Yes – active-active dual storage nodes per virtual

LUN

appliance may improve or reduce performance

Yes

Yes

Yes

Yes

Appliance-Based

Some solutions (e.g., Lefthand); potential to evolve

Yes

Yes – leading solutions employ n-way distributed

clustering

Yes

Yes

native disk drives only, but at commodity prices

Yes

Tied to single Array or cluster

Next Generation Virtual Array

Common Centrally-Managed Services

Policy-Based Automation

Non-disruptive Installation, Migration & De-Migration

Shared Virtual Name Space

Policy-Based Tiered Storage

No Practical Impact to Operational Performance

Heterogeneous Access & Sharing

Non-disruptive Infrastructure Provisioning

Meets Few If Any RequirementsMeets Some RequirementsMeets Most RequirementsMeets/Exceeds Requirements

26


Homogeneous Array Clustering as a Technology of Scale

� Sample players:► IBM – XIV

►3PAR

►Compellent

►Dell – EquaLogic

►Pillar Data

►HPQ - Left Hand

►XioTech - ISE

� Related Models►Sun Thumper

►EMC InfiniStore

27


Hardware architecture

� Commodity switches

act as a backplane

� Interface modules: commodity servers act as ‘unintelligent’ routers

� Data modules: commodity servers with SATA drives act as ‘smart’ storage sub systems:

► Caching

► Replication functionality

► Virtualization

� UPSs

28


Interface Module

Array Cluster Architecture Example - XIV

Interface Module

Interface Module

Internal Switch 1 Internal Switch 2

Data Module

Data

Data Module

Data

Data Module

Data

Host Host Host Host Host

User Switch (FC/Ethernet)

29


Use Case Scenarios of the Scale-out Architecture� Digital Archive

► Low cost, highly scalable storage needed for massive amounts of archive data that requires acceptable response time that is easy to configure and manage

� Web 2.0

► Low cost, highly scalable storage with predictable performance and quick response times for massive amounts of data uploaded and viewed by millions of people

► Need to easily add capacity on a regular basis and will not use FC SANs

� Service Providers

► Low cost, highly scalable storage for offsite data storage with advanced, efficient Snapshot technology for backups.

� Digital Media

► Low cost, highly scalable storage needed for digital streaming applications with predictable high sequential performance that is easy to configure and manage

� Development / Test

► Low cost, predictable high performance storage needed for development/test environments

► Advanced efficient Snapshot capabilities with no performance impact to production data required

� Clustered Computing

► Predictable, high sequential performance storage that scales with capacity with load balancing and no hot spots needed for large computational models in clustered server environments

30


Solid State Devices

31


Typical System Time Scale

Time

[ns]

CPU operations (1ns)

Get data from L2 cache (10ns)

102

Get data from DISK (5ms)

108

Get data from TAPE (40s)

Storage

Memory

103

104

105

106

107

109

1010

Get data from DRAM (60ns)

10

1 second

minute

hour

day

week

month

year

decade

century

Human scale

Write to FLASH, random (1 ms)

Read a FLASH device (20 us)

SCM

32


Storage Technology Summary

� Disk drives are the current technology►Very good $/Gbyte, but slow access performance

►Access performance is slow – 150-200 Ops/Sec

� NVRAMs appearing as contenders►Flash and DRAM hybrid SDDs available now

● Other Storage Class Memory technologies in the wing s► 20-60X more expensive in $/GB today

►50K Ops/sec Read, 17K Ops/sec write

●Up to 5X better in $/Op/sec●Much lower latencies for better application performan ce

►Can now fill niche for high Ops/Sec applications

►More general applications will require new technology

33


Candidate Device Technologies

�SSDs with DRAM only – Have been available for many yea rs�100X cost of disk or more, little historical use

�SSDs combining DRAM and Flash – Available now�SLC Flash now�MLC Flash in the future?

�FeRAM (Ferroelectric RAM)� FeFET

� MRAM (Magnetic RAM)� Racetrack memory

� RRAM (Resistive RAM)� Organic & polymer memory

� Solid Electrolyte� PC-RAM (Phase-change RAM)

34


Solid State Storage Implementation ModelsServer

Cache

DA

DISK

FLASHSSD

DA

DA

HA

HA

HA

DISK

DISK

DISK

DISK

ServerServer

FLASH

SSD

Optimized Solid State ControllerFLASH

FLASH

Server ServerServer

Server ServerServer

RAM

SolidState

I/O

RAM

SolidState

I/O

RAM

SolidState

I/O

Internal Server

• Highest individual solid state performance

• More limited for shared applications

Optimized Controller

• High performance external storage

• Highest total IO/sec• Response time improvement• Broader application usage

Emulated Drive in Existing Disk System

• Simplest to develop• Lowest performance• Limited capacity points• Valuable for targeted niche

applications

35


SSD Flash Storage SystemIBM Project Quicksilver Demonstration

Sustained IO/s

IBM SVC IBM Quicksilver

1,000,000

0

500,000IOP

S

HDDSPC1

362% faster

Quick

silver

Quick

silver

Up to 24 Quicksilver nodes

SAN connected hosts

4 x 4GbpsFC ports per node

Legacy HHDcontrollers

SANdata

migration

SVCSVC

Hursley (UK) Million IOPS Quicksilver Cluster (Lab configuration)

� IO Performance Results:► 7/21 achieved sustained 1M IOPS with random 4K IO s ize,

70%R/30%W with queue depth of 16. Average latency m easured in system level is 700usec.

� Host hardware► 2 p590 servers. (Each server has 26 cores and 28GB memory)

� Quicksilver Storage Cluster Hardware► 14 SVC Nodes

► 12 x3650 servers (Each server hosts 2 FusionIO Iodri ve 160GB SSDs)

► 17 x3550 servers (Each server hosts 1 FusionIO Iodri ve 160GB SSD)

► Total 41 FusionIO IODrive SSDs.

36


Tape: Average Storage Cost Trends

2003 2004 2005 2006 2007 2008

$0.10

$50.00

$/GB

Industry Disk

HC LC Disk

Average Tape

Source: Disk - Industry Analysts, Tape - IBM

37


Futures of Tape Technology

� Tape-based backup/restore challenged by disk based approach with Snap copy or CDP ► Disk-based systems offer faster restore times► Software backup to disk and VTLs

� Hybrid disk and tape use for archive applications, whi ch will show significant growth ► Lowest cost storage► Greenest

� IDC estimates that between 2006 and 2010 worldwide revenues for► Tape drives will shrink by -8.7% CAGR ► Tape automation will shrink by 1.5% CAGR

� IBM’s strategy – grow share and address growth markets

38


Storage Hardware Approach for Backup & Archives

Store 250TB with 25% Growth Rate over 10 Years

TS7500 Virtualization Engine for Open Systems

TS7700 Virtualization Engine for System z

DR550 and WORM Tape

N series and Tape

Tape Encryption

Blended disk and tape solutions best address customer goals

Customer Storage Goals:•Performance•Compliance

•Security•Data Protection

•TCO

10 Year TCO Analysis

39


Commercial Data Center SANsTechnologies and Topologies

� Fibre Channel is broadly used in Commercial Data Center

► SAN use growing

► iSCSI ramping

► IB storage?

� SANs typically are:

► Local, single fabric vendor

► Over provisioned

► Multiple SANs per DC● Some use of SAN routers

� Commonly used for distance copy

► Disaster recovery

► FC channel extension over IP

� Growing use of iSCSI as a low cost fabric, centralization technology for campus, or for cross enterprise distributed data

iSCSI & TOEDhip

HBA

iSCSI & TOEDhip

HBA

iSCSI & TOEDhip

HBA

iSCSI & TOEDhip

HBA

Integrated Integrated Monitoring and Monitoring and ManagementManagement

Campus Campus SystemsSystems

Ethernet LinksEthernet Links

Desktops & LaptopsDesktops & Laptops

Departmental Departmental ServersServers

Desktops & LaptopsDesktops & Laptops

iSCSI & TOEDhip

iSCSI & TOEDhip

Data Center Fabric

Disk and Tape Storage ControllersDisk and Tape Storage Controllers

iSCSI & TOEDhip

iSCSI & TOEDhip

Data Center Fabric


iSCSI & TOEDhip

iSCSI & TOEDhip

Data Center Fabric


40


Data Center Fabric Convergence �� Value Proposition

� Fabric Convergence Agenda

� Convergence Levels

� Today’s options

► InfiniBand

► iSCSI and NAS

� Emerging option

► FC over Convergence Enhanced Ethernet

Fabric Convergence

Servers Multiple Fabrics Converged Fabric

Improved RASReduced failure points, time, misconnections, bumpi ng.

Lower CostLess adapters, cables & switches

Simpler ManagementSingle physical fabric to manage.

Simpler to deploy, upgrade and maintain.

41


Convergence Enhanced Ethernet (CEE, as in Sea) Over view

� There is no protocol being created in IEEE 802 name d CEE.

► So what in the world is CEE?

� CEE is the name folks have given to the output spec ifications of the following 3 functions which are be ing standardized in IEEE 802:

► Priority processing and packet scheduling

► Per priority flow control (e.g. PAUSE)

► Discovery and capability exchange protocol

� Two additional convergence enhancements are being m ade to Enet, but EMC, HP and IBM view them as not required for FCoE.

► Backward Explicit Congestion Notification (BECN) an d TRILL.

CEETM

System System

Switch/Router

� LAN� Storage� Clustering

CEETM

� Streaming� Management

IBM IBM

Traffic Examples

FC Stack

HPC Stack

10 GigE PhysicalCEE Link

IP Stack

Middleware

Application

42


Scalable File Serving

43


Network Attached Storage (NAS) Directions� NAS major in simplicity, ease of use

► Appliances or Gateways

► Widely adopted standard protocols

● NFS, CIFS, TCP/IP► File based replication – local and WAN

► Networking security, File layer security

� The issues: Performance and scale► IOP performance

► One-box or partial box limit to a namespace

� Directions► NAS aggregators (NAS virtualization)

► Large scale NAS based on cluster file systems

► High performance interfaces

► New data protocols – attempts have failed to date

LAN

Internal Fabricor SAN

File Server

NFS / CIFS Clients

Storage

44


Compute Clusters with Scalable File Systems

App comm NetworkGigE, IB, Myranet, Federation… Remote

Systems/networks

Storage Storage Storage StorageStorage Storage

…

App Server

App Server

App Server

App Server

Cluster File System Configuration

IOServer IO

Server

IOServer

�Large scale cluster FS

�Shared namespace

�NFS/CIFS export

�Block/Object Store

�Systems vendors

�IBM

�Sun

�HP

�Quantum/ADIC

�Panasas

�EMC/Ibrix

�…

�Open Source

�Lustre

HPSS

Data

Mover

45


Sharing Data

� Example: grid computing workflows► Data is ingested from a

variety of input sources at remote locations

► Compute resources access input data and store their results

► Processed data is visualized by people at many locations

� … with everything potentially at different locations!

Ingest

Compute

Visualize

46


Cluster File Systems – Usage Topologies

Compute NodeStorage Node

NFS/CIFS Client

external fabric

NFS/CIFS clients

Compute cluster

internal fabric

NSD

storage servers

internal fabric

SAN

�Highly Scalable NAS

Compute cluster NSD

storage servers

47


Scale out NAS – An ExampleSimilar implementations of varying degrees of maturity found in other vendor’s products.

� General Parallel Filesystem (GPFS) -

IBM’s High end clustered file system

� CTDB – Clustered Trivial Database Daemon, Controls the cluster and the file service daemons

� Enhanced CIFS Server with NTFS Semantics to Support Active Directory Integration

� SoFS Package – Provides Management GUI, Apache file server module, acceleration tools, etc.

� IBM Hardware

48


Information Archiving

49


Archive Capacity by Media Type

0

5,000

10,000

15,000

20,000

25,000

30,000

2005 2006 2007 2008 2009 2010

Total Digital Archive Capacity, by Content Type - Worldwide (PB)

E-mail Database Unstructured

54%19,58011,5017,1804,7843,2722,295Unstructured

58%27,20615,0769,4016,0964,0812,786Total Capacity (PB)

79%4,0732,0621,102639386222Database

68%3,5531,5121,119673423269E-mail

CAGR201020092008200720062005

Source: ESG Research Report: Digital Archiving: EndESG Research Report: Digital Archiving: End --User Survey and Market Forecast 2006 User Survey and Market Forecast 2006 -- 20102010

50


Storage Archive Characteristics

� Capture and retain all relevant documents► Provide repository for multiple types of informatio n (files, databases, e-

mail)► Capture incoming and outgoing e-mail before unmodif ied by users► Verify recording integrity and accuracy► Protect from malicious or accidental deletion durin g the retention

period

� Provide options for retaining information► Provide non-erasable, non-rewritable compliance pro tection► Offer options for non-compliance data on erasable m edia► Support mirrored or duplicated storage► Migrate data to new media as old media wears out

� Enforce the disposal of information at the appropriate time, but not before

► Enforce retention periods and holds on all stored d ata and indexes► Delete immediately when information is no longer re quired for

regulatory, legal or corporate requirements

51


Archive Solution Stack

ContentManagement

Layer

ArchiveInfrastructure

Layer

ArchiveHardware

Layer

EnterpriseContent

Management

DatabaseArchives

HealthcareArchiveSolution

IndustryContent

Managers

CustomerDeveloped

ContentManagers

Disk StorageLow cost storage

De-duplicating storageHigh performance ultra dense storage

Tape StorageHigh areal density cartridges

High capacity tape framesWORM and Encryption Capabilities

Policy Based RetentionTiered Storage Hierarchy Management

Standard Ingest InterfacesIndexing and Search

Automated Data Migration for Long Term Data Preserv ation

Email&

Messaging

52


What is Data De-duplication?

� Data de-duplication (often called "intelligent compression") is a method of reducing storage needs by eliminating redu ndant data. Only one unique instance of the data is actua lly retainedon storage media. Redundant data is replaced with a reference or pointer to the unique data copy.

1. Data elements are evaluated to determine a unique signature for each

2. Signature values are compared to identify all

duplicates

3. Duplicate data elements are eliminated and are replaced with pointers to the existing

reference element

C

A

B

C

AA B

B

A C

A

B

C

AA B

B

A C

A

B

C

AB

B

A

A

53


IBM Research – Archive Architecture

Ingest Interfaces

Document ingest via a variety on standard file-based protocols.

Extended metadata can be passed by setting extended attributes or by storing a companion XML control file.

Any combination of ingest / retrieval is supported

Applications need not have knowledge of what tier the document resides.

Global namespace is provided.

InformationArchive

Manager

Remote IAMInstance

54


The Preservation Archive Challenge� These documents were created by pre-digital societies. The

information content is still accessible

Dead Sea Scroll, ~70AD.Media: Copper.

Language: Hebrew.

� This information was created a few years ago►Will the media last for 20 years? ►Will it be possible to access, interpret and presen t the

data in 20 years? 50? 100?

Mayan Glyph, Palenque ~630AD.Coronation of King Pacal

26 March, 603

55


The Two Faces of the Digital Archive Challenge

Bit Preservation Logical Preservation

How do you ensure that you can

retrieve a bit perfect copy of digital data

after years or decades?

How do you ensure that once you’ve retrieved the bit perfect copy, that you can productively use the data?

56


OAIS: Open Archival Information System Reference Model

Functional Model

�Provide fundamental ideas, concepts and a reference model for long-term archives

�Incorporate emulation, migration, descriptive via encapsulation

Information Model

57


PDS Architecture

� Layered approach based on open standards - OAIS, XAM, OSD

� In CASPAR, all layers are utilized. Other embodiments may use only some layers.

� Utilize XAM to provide logical abstraction of containers (XSets)

� Offload preservation functionality to the storage

ingestAIP, accessAIP,

transformAIP, getPreservationPolicy

58


Large Scale Storage Management

59


Management of Information Infrastructure

� End-end management of Information Infrastructure is key to business value

� Many categories involved:► Performance

► Availability

► Security

► Workflow

► Data replication

► Asset Management

► Power

► …


Backup/HSM

File Systems

Content Management


Storage Devices

DBMS

Applications


Storage M

anagement

Security M

anagement

Archive Replicate

System

s Managem

ent

60


But what if your infrastructure looks like this?

61


� Businesses spend a large fraction of their IT budge ts on data center resource management rather than on valuable applications and business processes

� Data center complexity has reached record levels an d is continuing to increasethereby limiting IT improvements and benefits

Topologies of federated services must be mapped ont o large numbers of diverse physical and virtual resou rces

Sea of Heterogeneous Servers, Storage, Networks and Their Virtualization

Business Processes as Services

62


Data Center “Ensemble” Approach To IT Simplification

Topologies of federated services are mapped onto sm aller numbers of systems.Ensembles replace ad hoc groups of systems.

Business Processes as Services

Sea of Heterogeneous Servers, Storage, Networks and Their Virtualization

EnsembleEnsemble

EE

� An ensemble is pool of like systems that are managed as a singl e system► It consists of a pool of compatible networked syste ms with integrated virtualization and

management software that manages and leverages virt ualization

► It can scale from a few to many thousands of server s, while having management complexity / cost essentially independent of its size and like that o f a single system

► Parallel Sysplex, Blue Gene, and Google are example s of ensembles in many ways

► Virtual resource mobility is an important capabilit y within most ensembles

Over time, the use of ensembles

will increase

63


Service Orientation, CloudServices, Integration, …

Top IT Requirements (all are vital)

• Agility – rapid deployment, self-service, …

• Resiliency – availability, disaster recovery, …

• Security – trusted computing, surveillance, …

• Greenness – energy efficiency, low impact, …

• Low Cost – TCO (HW, SW, labor, facilities, …)

Abstractionand Pooling

Multi-System Virtualization

Virtual Servers, Storage, Networks

Storage

Servers

Networks

V

V

V

Scale-OutSprawl

Windows Servers

Linux Servers

Unix Servers

ManagementServers

Switches

Storage

Firewalls,Routers

PhysicalConsolidation

WindowsServer

Linux Server

Mainframe orUnix Server

Networks

Storage

V

VV

V

V

IT Simplification

Ensemble

Ensemble

Ensemble

New Enterprise Data Center

Key Technologies (unordered)

• Service oriented architecture• End-to-end service mgmt• Comprehensive virtualization• Ensembles & scalable servers• Converged networks• Cloud computing services

• Software as a service• Information as a service• IT appliances• Real-time data streams• Mobile client services• Virtual worlds

64

© 2008 IBM CorporationStorage Ensembles

� General purpose Enterprise Archive with Content management

� App specific archive – email� Industry specific archives –

medical images

Archive

Stores

� Large scale NFS and CIFS data access

� Stream servers� Compliance stores

� SAN storage as an (enterprise ) service

� Building block for larger ensembles

� Data Warehouse cluster

� High performance OLTP

ExamplesStorage Ensemble

Database

File

Level

Block(LUN)

Single System

Block Storage Ensemble

File Server Ensemble

Block Storage Server

File Server

Database Server/Storage ClusterDatabase Server

Xaction serviceQuery serviceBackupRemote copy

Cluster FSFile APIsQOSStorage TiersBackupRemote copyArchive interface

Cluster FSFile & Archive APIsIndexing/SearchRetention PolicyComplianceQOSStorage Tiers incl TapeBackupRemote copy

IntegrationProvisioningQOS, load balanceVirtual storageAutomated mgmtDisaster recovery

SAN ComponentsSAN Ensemble

Cluster FS

Cluster FSContent Mgmt

65


Thank-You!

Storage Industry Overview for the Disney Architecture · PDF fileStorage Industry Overview for...

Documents

Transcript of Storage Industry Overview for the Disney Architecture · PDF fileStorage Industry Overview for...