San School Book

7/29/2019 San School Book

1/71

STORAGE ARCHITECTURE/

GETTING STARTED:SAN SCHOOL 101Marc Farley

President of Building Storage, Inc

Author, Building Storage Networks, Inc.


2/71

Agenda

Lesson 1: Basics of SANs

Lesson 2: The I/O path

Lesson 3: Storage subsystems

Lesson 4: RAID, volume managementand virtualization

Lesson 5: SAN network technology

Lesson 6: File systems


3/71

Basics of storage networking

Lesson #1


4/71

Router

Concentrator

Bridge

Dish

Network Switch/hub

Computer

System

HBA or

NIC

VPN

Connecting


5/71

Networking or bus technology

Cables + connectors

System adapters + network device drivers

Network devices such as hubs, switches, routers

Virtual networking

Flow control

Network security

Connecting


6/71

Tape

Drives

Disk

Drives

RAID

Subsystem

Volume

Manager

Software

Mirroring

Software

(wiring, network transmission frame)

Storage Command and Transfer Protocol

Storage

Device

Drivers

HostSoftware

Storage

Protocol

Storage

Devices

Storing


7/71

Device (target) command and control Drives, subsystems, device emulation

Block storage address space

manipulation (partition management) Mirroring

RAID Striping

Virtualization

Concatentation

Storing


8/71

User/Application View

C:\directory\file

User/Application View

Database

Object

(Storage)

Logical Block Mapping

Filing


9/71

Namespace presents data to end users andapplications as files and directories (folders)

Manages use of storage address spaces

Metadata for identifying data

file name

owner dates

Filing


10/71

Connecting, storing and filing as a

complete storage system

Computer System

HBA or

NIC

Network Switch/hub

Disk Drive

Wiring StoringFiling

Storing function in

an HBA driver

Cable Cable

Connecting


11/71

NAS and SAN analysis

NAS is filing over a network

SAN is storing over a network

NAS and SAN are independent technologies

They can be implemented independently

They can co-exist in the same environment

They can both operate and provide services to the same

users/applications


12/71

Protocol analysis for NASand SAN

Storing

Wiring

FilingNAS

SAN

Network

Filing

Connecting

Storing


13/71

Storing

Wiring

Filing

NAS Head

ServerSystem

NAS

Client

Wiring

SAN

Storage

NAS Server

+SAN Initiator

NAS HeadSAN

Target

Connecting

Storing

Connecting

Filing

Integrated SAN/NAS environment


14/71

Storing

Wiring

Filing

NAS Head

ServerSystem

NAS

ClientSAN

Storage

SAN

Target

Storing

Connecting

Filing

NAS Head

Common wiring with NASand SAN


15/71

The I/O path

Lesson #2


16/71

Host hardware path components

Processor MemoryBus

SystemI/O Bus

Storage

Adapter(HBA)

Memory


17/71

Host software path components

ApplicationOperatingSystem

FilingSystem

VolumeManager

Device

Driver

Multi-Pathing

CacheManager


18/71

Network hardware path components

Cabling

Fiber opticCopper

Switches, hubs, routers, bridges, gatways

Port buffers, processors

Backplane, bus, crossbar, mesh, memory


19/71

RoutingFabricServices

Virtual

Networking

Access and

SecurityFlow

Control

Network software path components


20/71

Network

Ports

Access and

SecurityInternal Busor Network

Cache Resource

Manager

Subsystem path components


21/71

Disk drives Tape drives Solid state devicesTape Media

Device and media path components


22/71

The end to end I/O path picture

Disk

drives

Tape

drives

Network Systems

Access

and

Security

Internal Bus

or NetworkCache

Cabling RoutingFabric

Services

Virtual

NetworkingAccess and

Security

ProcessorMemory

Bus

SystemI/O Bus

Storage

Adapter

(HBA)

MemoryApp Operating

System

FilingSystem

VolumeManager

Device

DriverMulti-

Pathing

CacheManager

Subsystem

Network PoirtResource

Manager

Flow

Control


23/71

Storage subsystems

Lesson #3


24/71

StorageResources

Generic storage subsystem model

NetworkPorts

Cache Memory

Controller (logic+processors)

Access control

Resource manager

Internal Bus

or Network Power


25/71

Redundancy for high availability

Multiple hot swappable power supplies

Hot swappable cooling fans

Data redundancy via RAID

Multi-path support Network ports to storage resources


26/71

Physical and virtual storage

Physicalstorage

device

Physicalstorage

device

Physicalstorage

device

Physicalstorage

device

Subsystem Controller

Resource Manager

(RAID, mirroring,

etc.)

Exported

storage

Exported

storage

Exported

storage

Exported

storage

Exported

storage

Exported

storage

Exported

storage

Exported

storageHotSpare

Device


27/71

SCSI communications are independent of connectivity

SCSI initiators (HBAs) generate I/O activityThey communicate with targets

Targets have communications addresses

Targets can have many storage resources

Each resource is a single SCSI logical unit (LU) with a universal

unique ID (UUID) - sometimes referred to as a serial number An LU can be represented by multiple logical unit numbers (LUNs)

Provisioning associates LUNs with LUs & subsystem ports

A storage resource is not a LUN, its an LU

SCSI communications architecturesdetermine SAN operations


28/71

Provisioning storage

Physicalstorage

devices

Physicalstorage

devices

Physicalstorage

devices

SCSI LU

UUID B

SCSI LU

UUID A

SCSI LUUUID C

SCSI LUUUID D

Port S4

Port S3

Port S2

Port S1

LUN 0

LUN 1

LUN 1

LUN 2

LUN 2

LUN 3

LUN 3

LUN 0

Controller functions


29/71


30/71

Exported

Volume

Exported

Volume

Exported

Volume

Exported

Volume

Controller Cache Manager

Read Caches

1. Recently Used

2. Read Ahead

Write Caches

1. Write Through (to disk)

2. Write Back (from cache)

Caching


31/71

Tape Subsystem Controller

Tape SlotsRobot

Tape subsystems

TapeDrive

TapeDrive

TapeDrive

TapeDrive


32/71

ExportedStorageResource

Management station

browser-based

network mgmt software

Ethernet/TCP/IP

Out-of-band management port

Storage SubsystemIn-band

management

Now

withSMIS

NowwithSMIS

Subsystem management


33/71

Duplication

Parity

Difference

2n

n+1

-1

d(x) = f(x) f(x-1) f(x-1)

Data redundancy


34/71

I/O PathMirroringOperator

I/O PathAA

I/O PathBBTerminate I/O & regenerate new I/Os

Error recovery/notification

Host-based Within a subsystem

Duplication redundancy with mirroring


35/71

Host

Uni-directional

(writes only)

A BAAA

Duplication redundancy with remote copy


36/71

Subsystem Snapshot

Host

A BAAA C

Point-in-time snapshot


37/71

Lesson #4

RAID, volume management

and virtualization


38/71


39/71

Late 1980s R&D project at UC Berkeley David Patterson

Garth Gibson

(independent)

Redundant array of inexpensive disks Striping without redundancy was not defined (RAID 0)

Original goals were to reduce the cost and

increase the capacity of large disk storage

History of RAID


40/71

Capacity scaling Combine multiple address spaces as a single virtual

address

Performance through parallelism Spread I/Os over multiple disk spindles

Reliability/availability with redundancy Disk mirroring (striping to 2 disks)

Parity RAID (striping to more than 2 disks)

Benefits of RAID


41/71

RAID

Controller(resource

manager)

Storage

extent 1

Exported

RAID

disk

volume

(1 address)

Storage

extent 2

Storage

extent 3

Storage

extent 4

Storage

extent 5

Storage

extent 6

Storage

extent 7

Storage

extent 8

Storage

extent 9

Storage

extent10

Storage

extent11

Storage

extent12

Combined extents1 - 12

Capacity scaling


42/71

RAID controller (microsecond performance)

Diskdrive

Diskdrive

Diskdrive

Disk

drive

Disk

drive

Disk

drive

Disk drives (Millisecond performance)

from rotational latency and seek time

12 3 4 5

6

Performance


43/71

RAID arrays use XOR for calculating parity

Operand 1 Operand 2 XOR ResultFalse False FalseFalse True TrueTrue False TrueTrue True False

XOR is the inverse of itself

Apply XOR in the table above from right to left

Apply XOR to any two columns to get the third

Parity redundancy


44/71

Reduced mode operations

When a member ismissing, data that isaccessed must be

reconstructed with xor

An array that isreconstructing data is saidto be operating in reduced

mode

System performanceduring reduced modeoperations can besignificantly reduced

XOR {M1&M

2&M

3&P}


45/71

RAID Parity RebuildThe process of recreating data on a replacement memberis called a parity rebuild

Parity rebuilds are often scheduled for non-production

hours because performance disruptions can be so severe

XOR {M1&M

2&M

3&P}

Parity rebuild


46/71

Hybrid RAID: 0+1

Diskdrive

Diskdrive

Diskdrive

Diskdrive

1 2 3 4 5

Diskdrive

Diskdrive

Diskdrive

Diskdrive

Diskdrive

Diskdrive

Mirrored pairs of striped members

RAID 0+1, 10

RAID Controller


47/71

Volume management and virtualization

Storing level functions

Provide RAID-like functionality in host systems

and SAN network systems

Aggregation of storage resources for:

scalability

availability

cost / efficiency

manageability


48/71

RAID & partitionmanagement

Device driver layer

between the kerneland storage I/Odrivers

OS kernel File system

Volume Manager

HBA drivers

HBAs

Volume Manager

Volume management


49/71

SAN disk

resources

Volume

manager

Virtual

Storage

Server system

HBA drivers

SAN HBA

SCSI BusSCSI HBA

SCSI disk resource

SAN SwitchSAN cable

Volume managers can use allavailable connections and

resources and can span multipleSANs as well as SCSI and SAN

resources


50/71

RAID and partition management in SAN systems

Two architectures:

In-band virtualization (synchronous)

Out-of-band virtualization (asynchronous)

SAN storage virtualization


51/71

Disk

subsystems

SANvirtualizationsystem

Exported virtual storage

I/O Path

System(s),

switch or

router

In-band virtualization


52/71

Distributed volumemanagement

Virtualization agentsare managed from acentral system in theSAN

Virtualizationagents

Disk

subsystems

Virtualizationmanagementsystem

Out-of-band virtualization


53/71

Lesson #5

SAN networks


54/71

The first major SAN networking technology

Very low latency

High reliability Fiber optic cables Copper cables

Extended distance

1, 2 or 4 Gb transmission speeds

Strongly typed

Fibre channel


55/71

A Fibre Channel fabric presents a consistent interfaceand set of services across all switches in a network

Host and subsystems all 'see' the same resources

SAN

TargetStorage

Subsystem

SAN

TargetStorage

Subsystem

SAN

TargetStorage

Subsystem

Fibre channel


56/71

FC ports are defined by their network role

N-ports: end node ports connecting to fabrics

L-ports: end node ports connecting to loops NL-ports: end node ports connecting to fabrics or loops

F-ports: switch ports connecting to N ports

FL-ports: switch ports connecting to N ports or NL ports in

a loop E-ports: switch ports connecting to other switch ports

G ports: generic switch ports that can be F, FL or E ports

Fibre channel port definitions


57/71

Ethernet / TCP / IP SAN technologies

Leveraging the install base ofEthernet and TCP/IP networks

iSCSI native SAN over IP

FC/IP FC SAN extensions over IP


58/71

Native storage I/O over TCP/IP New industry standard

Locally over Gigabit Ethernet

Remotely over ATM, SONET, 10Gb Ethernet

iSCSI

TCP

IP

MAC

PHY

iSCSI


59/71

Storage NICs (HBAs)

SCSI drivers

Cables Copper and fiber

Network systems Switches/routers

Firewalls

iSCSI equipment


60/71

FC/IP

Extending FC SANs over TCP/IP networks

FCIP gateways operate as virtual E-port connections

FCIP creates a single fabric where all resources appear

to be local

FCIPGateway

FCIPGateway

FCIP

Gateway

FCIP

Gateway

TCP/IP

LAN, MANor WANE-port E-port

One fabricOne fabric


61/71

SAN switching & fabrics

High-end SAN switches have latencies of 1 - 3

sec

Transaction processing requires lowest latency Most other applications do not

Transaction processing requires non-blockingswitches

No internal delays preventing data transfers


62/71

Switches 8 48 ports

Redundant power supplies Single system supervisor

Directors 64+ ports

HA redundancy

Dual system supervisor

Live SW upgrades

Switches and directors


63/71

Star

Simplest

single hop

Dual star

Simple network

+ redundancy

Single hop

Independent or integrated

fabric(s)

SAN topologies


64/71

N-wide star Scalable

Single hop

Independent or

integrated fabric(s)

Core - edge Scalable

1 3 hops

integrated fabric

SAN topologies


65/71

Ring

Scalable

integrated fabric

1 to N2 hops

Ring + Star

Scalable

integrated fabric

1 to 3 hops

SAN topologies


66/71

Lesson #6

File systems


67/71

Name space

Access control

Metadata

Locking

Address space management

File system functions


68/71

StoringStoringFilingFiling


69/71

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 1819 20 21 . . . . 26 27

28 29 . . . . . 35 36

37 38 . . . . . 44 45

46 . . . . . . 53 54

55 . . . . . . 62 6364 . . . . . . 71 72

73 . . . . . . 80 81

82 83 84 85 86 87 88 89 90

Think of the storage address space as a sequenceof storage locations (a flat address space)


70/71

SuperblocksSuperblocks are known addresses used to findfile system roots (and mount the file system)

SB

SB


71/71

Filing and Scaling

StoringStoringStoringStoringFilingFiling

1 2 3 4 5

6 7 8 9 10

11 12 13 14 15

16 17 18 19 20

21 22 23 24 25

1 2 3 4 5 6

7 8 9 10 11 12

13 14 15 16 17 18

19 20 21 22 23 24

25 26 27 28 29 30

31 32 33 34 35 36

37 38 39 40 41 42

File systems must have a known anddependable address space The fine print in scalability - How does the filing function

know about the new storing address space?

San School Book

Documents

Transcript of San School Book