Designing Scalable SAN using MDS 9396S

using Cisco MDS 9396SDesigning Scalable SANParesh Gupta, [email protected]

Technical Marketing Engineer, Cisco

August 2015

mailto:[email protected]

2© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Multi-Protocol Product Portfolio: SAN, LAN, and Compute

10+ Years of Proven NX-OS Operating System Cisco Prime Data Center Network Manager (DCNM)

CONSISTENT AND SIMPLIFIEDFeatures, Management, and Programmability

Cisco UCS C-Series

Rack Servers

Cisco UCS B-SeriesBlade Servers

Cisco UCS Fabric

InterconnectsCisco UCS

6248UP

Cisco UCS 6296UP

LAN/SAN

Cisco Nexus 9000

Cisco Nexus 7000

Cisco Nexus 5600

Cisco Nexus 5500

CiscoNexus 3000

CiscoNexus 2000

Cisco MDS 9700

48x16G Line-Rate

FC

Cisco MDS 9250i

Cisco MDS 9148S

48x10G Line-Rate

FCoE

SAN 16G COMPUTE

Cisco MDS 9396S

July2015


Methodology for a continuous life-cycle of a SAN

Prepare

Plan

Design

Implement

Operate

Optimize

Storage Area Network


Propose an architecture to meet business requirements and goals

Preparation phase Prepare

Top quality user experience and application performance

Maximize availability (Reduced downtime)

Lower TCO (Reduced CapEx & OpEx)

Expandable architecture to support future business growth

Define Business

Goals

Create business plan with cost-benefit analysis


Gather network requirements, existing gap analysis and project plan

Planning phase

Human resource planning

Environmental planning (Power, Real State, etc)

Gap analysis, Risks and Constraints

Project management, timeline and mile stones

Create Project Plan

Sourcing options (Vendor selection)

Plan


MDS 9396S achieves your business goals

Prepare Plan


• Best of both worlds: Performance of a Director in a fabric switch!• 96 ports at line rate 16 Gbps Fibre Channel in compact 2 RU• Non-blocking, non-oversubscribed architecture• Consistent and Predictable performance across all ports• No local switching dependency and restrictions

Top Quality User Experience & Application Performance

MDS 9700

48 port, 16Gbps FC line card

MDS 9396S

delivered by

High Performance of

MDS 9396S


• Two hot swappable fan trays• Choice of reversible airflow• Port side exhaust at FCS

• 2 hot swappable power supplies

• 1+1 grid redundancy

• In Service Software Upgrade• 16 member ports in single port-channel (Link Aggregation)

• High Availability is achieved by spreading members across different ASIC, port-groups, etc

• Add or remove links as needed without traffic loss• Dual Redundant Power Supply (for Grid Redundancy)• Dual Redundant Fan Trays

Maximum availabilitydelivered

by

Robust Architecture of

MDS 9396S


• Base configuration: 48 ports or 96 ports• 48 port base can be upgraded with 12 port on demand license• Optional Licenses :

• Enterprise License for Extended B2B credits, QOS, IVR, VSAN based access control

• DCNM License for Fabric Management • DCNM + Enterprise license bundle

Expandable architecture: Support future business Growthdelivered

by

Flexible Licensing

48 Ports

12 Ports Increment

12 Ports Increment

12 Ports Increment

12 Ports Increment


• Leadership across Collaboration, Security, High-End Routing, Switching, Wireless, Cloud and Data Center Verticals, not just point products

• Leadership across LAN, SAN & Compute in Data Center• SAN: Industry partnership and Ecosystem• SAN: Strong Investment Protection, backward compatibility

Solution provider, not just a point product vendordelivered

by

Cisco’s ability to develop and support complete enterprise

IT infrastructure

Programmable: Integration with Industry-Leading Cloud Platforms

EMC ViPRSoftware Defined Storage

OpenStack

Microsoft System Center VMM

IBM PowerVC

Automate Zoning for faster application

deployments

User-Centric Models for Cloud-Scale Workloads

Cisco MDS 9000 Family: Extensive Industry Partnerships

Cisco Channel Partners MDS 9300/9200/9100 Only

Other Flash Vendors

Storage Array interop only

The Power of Ecosystem

Vblock

FlexPod HDS UCP Select

VSPEX

VersaStack

UCS for Red Hat

OpenStack

SmartStack

Pre-Packaged Converged Infrastructures


Tight integration between Cisco UCS and MDS 9396S

F-Port Channel

VSAN Trunking

F-Port Trunking for Multi-Tenancy

• Servers log in & participate in separate VSANs • Any UCS blade can connect over any N-Port uplink across

which the VSAN is trunked

F-Port Channel for Resiliency

• Port Channel between UCS & MDS: No Port-Pinning • No host relogin on individual link failure in Port Channel• No Manual re-balancing when new link added or removed

Operational Simplicity: NX-OS, DCNM

• Common operating system for MDS & UCS Fabric Interconnect (FI)

• Visibility of UCS FI and MDS in DCNM• Cisco TAC as common support system

MDS 9396S

UCS FI

Cisco UCS and Cisco MDS 9000 Family: Better Together White Paper

http://www.cisco.com/c/en/us/products/collateral/storage-networking/mds-9700-series-multilayer-directors/white-paper-c11-734381.html


Designing a network to meet Business and technical Requirements

Design phase Design

High Availability

Scalability

Security

Manageability

Design the network

Flexibility


Designing small single-switch SAN using MDS 9396S

Consolidation of SAN Islands

E-mailDatabase

BackupE-mail E-mailDatabase

Start with 48 port base license with single VSAN for

E-Mail Servers

Add 12 port license to accommodate Database

servers

Add another 12 port license to add Backup servers

MDS 9396S

Minimizes no. of SAN switches Centralized management

Blue VSANRed VSANYellow VSAN

Reduced TCO with best user experience & application performanceSecure & Flexible Design

48 – 96 Ports


Designing small multi-switch SAN using MDS 9396S

Reduced TCO with support for business expansionSecure & Flexible Design

MDS 9396S MDS 9396S

• Deploy in N-Port Virtualization (NPV) or N-Port ID Virtualization (NPIV) mode to reduce number of managed switches

• Maximize localization of servers and Storage on single switch: Reduced ISL traffic

BackupE-mail

MDS 9396S MDS 9148S

E-mail Database

12 – 192 Ports


Designing enterprise class SAN using MDS 9396S

Reduced OpEx & CapEx by reduction in no. of edge switches, power, cooling &cabling requirements with readiness for future growth

Massive Scalable Design

• MDS 9396S as Middle of Row (MoR) switch to consolidate server connections

• Even reduced number of managed devices with MDS 9396S as NPV & MDS 9700 as NPIV

• Quality of Service (QoS) for traffic prioritization

MDS 9700

MDS 9396S

MDS 9396S

MDS 9700

Edge-Core Architecture – Consolidation at Access Layer

1000s of Ports


SAN Extension using MDS 9396S

Business Continuity and Disaster RecoveryDesign to connect distant SAN islands

• Up to 25 KM: Use of CWDM optics at 8G or ELW optics at 16G• 512 KM @ 16G FC with DWDM infrastructure (Up to 4095 B2B credits per port with

enterprise license)

Primary Site

MDS 9396S MDS 9396S

MDS 9250i MDS 9250i

Native Fibre Channel basedMAN / WAN

IP MAN / WAN for FCIP

BackupSite

Disk replication, Tape backups

FC TrustSecEncryption

IPSecEncryption



Prepare Plan Design


Quick provisioning of network as per design

Implementation phase Implement

Connect the devices as per design

Configure the devices

Validate the configuration and connectivity

Training of support personals

Implement the Design

Quick & automated implementation to minimize time to deploy


• Intuitive approach to configure and Manage a Single Switch

• Point browser to the management IP address of the switch to access Device Manager

Cisco Device Manager

Watch Full demo (and many other videos) at youtube.com/ciscomds9000


• Unlike LAN switches, SAN switches require zoning for Servers and Storage Array to communicate.

• Quick Config Wizard configures zoning within minutes

Quick Configuration Wizard

Expedite Return on Investment


Quick Configuration Wizard


As number of switches increase, Configuring Single switch at a time is not a viable option


Power on Auto Provisioning (POAP) on MDS 9396S

Rapid Error Free Deployment

• Automation• Consistency

• ‘Go-Live’ time in minutes• No Human Errors

• Accuracy• Reliability

MDS 9396S…………………………………………………………………..

Provides:• IP Address, Gateway• Software Image• Configuration File


Maintain Network Health to support business operations

Operating a SAN Operate

Provisioning of new Servers & Storage

Performance Monitoring

Health Monitoring

Incident Management

Day-to-Day Operations

Fault Detection



Prepare Plan Design Operate


• Options of zoning via Intuitive NX-OS CLI

• Zoning can be done easily via DCNM

• Zoning Automation available via Industry leading Platforms: Cisco UCS Director, EMC ViPR, Microsoft System Center VMM, IBM Power PC, OpenStack.

Simplified & Efficient Zoning operations using MDS 9396S


Cisco Enhanced zoning on MDS 9396S

Secure and Resilient OperationsEnhanced Zoning

Serializes Configuration to Streamline Parallel Config Attempts from Multiple Users

HOST1

Users Acquire Fabric Lock One at a Time to Make Zone Config Changes

TARGET1

TARGET2

TARGET3

SWITCH1

SWITCH3

SWITCH2

SWITCH4

HOST2

HOST3

HOST4

TARGET4


• 8 Initiators & 4 targets in single zone create 132 entries (12 x (12-1)) within a SAN switch

• Ideal number of entries = 32

• SAN admins pay the price on internal inefficiency by creating 2-member zones. Too much of overhead for day-to-day operation!

• Solution: Cisco Smart Zoning

• Create single zone with all initiators and targets. MDS 9396S is smart to create just ideal number of entries (32)

Cisco Smart Zoning on MDS 9396S

Simpler, Efficient, IntuitiveSmart Zoning

I

T

II

I

I

I

I

I

T

T

T


Visualizing the benefit of Smart Zoning on MDS 9396SSmart ZoningRequires only 1 zone with 12 hosts and 2 X-Brick pwwns.

Zoneset name Smart-VDI vsan 200 zone name x-brick1 vsan 200 pwwn 15:10:10:10:10:10:10:01 initiator pwwn 15:10:10:10:10:10:10:02 initiator pwwn 15:10:10:10:10:10:10:03 initiator pwwn 15:10:10:10:10:10:10:04 initiator …….. pwwn 15:10:10:10:10:10:10:12 initiator pwwn 25:20:20:20:20:20:20:01 target pwwn 25:20:20:20:20:20:20:02 target

Traditional Zoning(Requires 24 two-member zones)Zoneset name VDI vsan 100 zone name host1-x-brick1a vsan 100 pwwn 10:10:10:10:10:10:10:01 pwwn 20:20:20:20:20:20:20:01 zone name host1-x-brick1b vsan 100 pwwn 10:10:10:10:10:10:10:01 pwwn 20:20:20:20:20:20:20:02 zone name host2-x-brick1a vsan 100 pwwn 10:10:10:10:10:10:10:02 pwwn 20:20:20:20:20:20:20:01 zone name host2-x-brick1b vsan 100 pwwn 10:10:10:10:10:10:10:02 pwwn 20:20:20:20:20:20:20:02 . . . . . . . zone name host12-x-brick1a vsan 100 pwwn 10:10:10:10:10:10:10:12 pwwn 20:20:20:20:20:20:20:01 zone name host12-x-brick1b vsan 100 pwwn 10:10:10:10:10:10:10:12 pwwn 20:20:20:20:20:20:20:02

Adding another VDI cluster would require an additional 72 zones to be configured per fabric.

Adding another VDI cluster would require adding the new storage ports and new hosts to the existing zone or adding the 2 new storage pwwns to the first zone and creating one additional zone with all 4 storage ports together with the new hosts.


Resilient Operations on MDS 9396S Operate

• Faulty equipment, loose SFPs, dirty or damaged cables, can result in packets getting corrupted

• CRC checking drops corrupted frames from end devices or internally corrupted frames

• FEC corrects frames corrupted in-flight to preserve frames

Ingress CRC

checking

Internal CRC

checking

Forward Error

Correction

Drop frame

Drop frame

CRC Detection and 16 Gbps FC Forward Error Correction (FEC)


MDS 9396S for Self Healing SAN

Misbehaving device “slowly drains” frames destined to it. Unrelated flows and devices get affected MDS auto-detects slow drain devices and takes recovery actions

Potential Reasons for Slow-Drain Devices• Speed Mismatch• HBA issues • Server Performance issues – application/OS• Non graceful Virtual Machine exit

Affected Host

Misbehaving HBA

Detection

2.5 µs

Automatic Recovery

Immediate

Troubleshooting

DCNM Slow Drain Diagnostics


Detecting and Troubleshooting Slow Drain using DCNM

Automates Troubleshooting

Reduces False Positives

Shows Fluctuations in Counters

Watch Full demo (and many other videos) at youtube.com/ciscomds9000


Programmability using MDS 9396S

OpenStack Integration

Full benefits of OpenStack along with seamless zoning

Embedded Event Manager (EEM)

NX-OS infrastructure to take user defined events on actions

Native on-switch TCL Interpreter

Near-Infinite possibilities with your own custom script

Scheduler

Capability to schedule any task on the switch (like Linux cron)


Programmability Use Case: Config Checkpoint

Config backup at remote server after each change

Switchname_Timestamp.cfg

Functionality

Behind the Scene

EEM catches config change

event

Triggers TCL script on switch bootflash

Script copies running_config to

remote server


Maximize the performance with proactive management

Optimizing a SAN

Network Capacity Trending

Network Health Trending

Aim to improve uptime

Aim to reduce number of incidents

Day-to-Day Operations

Take all the learnings to Prepare Phase

Optimize


MDS 9396S Health Score on DCNM

HEALTH SCORETRENDING

Identify and Remediate

HEALTH SCORETRENDING

IF YOU CAN MEASURE IT, YOU CAN FIX IT80% Health 60 % Health

Policy-Based Health Index

Quickly Determine Level of Risk

View Alerts Contributing to Low Score

Measure Success Overtime

Restore Health back Normal

MDS 9396S MDS 9396S



Prepare Plan Design OperateImplement Optimize


• Current Infrastructure unable to support user growth and data explosion• Planning for Access Network Consolidation to reduce CapEx• Already have large UCS install base and growing• Due to pressure on OpEx: Need for Programmability, Automation and Manageability• Unhappy with long history of Fork Lift upgrades with every speed change.

Business Goals• Best in class user experience: Support high demanding application (Today & Future) • Investment Protection for 10+ years• Minimize CapEx and OpEx• Consolidated global IT strategy and architecture

Design under exploration: Large 16 Gbps FC customer

Prepare

Plan


Design under exploration

MDS 9396S

MDS 9706

MDS 9710

. . . . . 80

• MDS 9706 as Storage Edge

• MDS 9710 as Core (Logically Server Edge)

• MDS 9396S as Top of Row (TOR) switch shared among servers at multiple racks

• Single rack has 20 – 30 servers

• F-Port Channel between NPV MDS 9396S & MDS 9710 NPIV for high availability

• Single NPIV MDS9710 will be attached to 80 NPV MDS 9396S in a single fabric

• POAP to provision hundreds of MDS 9396S

Design


Customer meets Business Goals using Cisco MDS

MDS 9396S

MDS 9706

MDS 9710

. . . . . 80

Best in Class User Experience

High performance, non-oversubscribed,

no-local switching

Cisco MDS as choice for multiple data centers across globe along with Cisco Nexus and UCS

Consolidated global IT strategy

Large CapEx saving by using MDS 9396S as Server Edge. OpEx saving by POAP, DCNM

Minimize CapEx & OpEx

MDS 9700 ready for 32 G FC & 40 G FCoE today. ISLs can easily be upgraded without forklift

Investment Protection

By high performing, non-oversubscribed, predictable, consistent, no-local switching architecture

Best in Class User Experience


• Existing Infrastructure unable to support user growth and data explosion• Existing infrastructure has shown inconsistent and unpredictable behavior under load• Existing infrastructure known to forward corrupt frames causing fabric wide disruption

Business Goals• Increase efficiency of VDI application running at Point of Sale (POS) to reduce time for

each customer• Stores located every few KMs. Maintain backup at central Data Center • Reduced CapEx and OpEx• Capacity for future expansion

Design under exploration: Flash Storage for VDI

Prepare

Plan


Flash Storage for VDI using MDS 9396S

• MDS 9396S as SAN switch between VDI application servers and Flash Storage

• Predictable & Consistent Performance provides freedom to connect Storage & Servers to any port on the switch without worrying local-switching complexity.

• Ingress CRC checking & dropping of corrupt frames will limit failure domains just to the affected device

• Operational simplicity by Smart zoning

• End-to-End visibility (VM & LUN awareness) by DCNM.

• MDS 9396S for long distance FC native synchronous replication over dark fiber to central data center

Design

Access VSAN

Replication VSAN

Dark Fibre to central Data Center – Continuous backup of

store operations

FC TrustSec Encryption


MDS 9396S achieves Business Goals for Flash StorageBest in Class User

Experience

High performance, non-oversubscribed,

no-local switching

By unmatched licensing flexibility of MDS 9396S. Customer starts with 48-port base & increments with 12-ports on-demand license

Capacity for Future

Expansion

By consolidation of Access & Replication SAN on same switch using Cisco VSANs. OpEx saving by DCNM

Minimize CapEx & OpEx

By synchronous replication between store & Central Data Center. Long distance FC link possible due to large number of B2B credits on MDS 9396S

Backup at Central Data

Center

By high performing, non-oversubscribed, predictable, consistent, no-local switching architecture of MDS 9396S. Cisco MDS + UCS + EMC XtremeIO has been validated for > 1M IOPS

Increased VDI Application

Performance

Access VSAN

Replication VSAN

Dark Fibre to central Data Center – Continuous backup of

store operations

FC TrustSec Encryption


Summary


Prepare

Plan

Design

Implement

Operate

Optimize

MDS 9396S

Coordinated Planning and Strategy

Assess Readiness

Design the Solution

Implement the Solution

Maintain Network Health

Operational Excellence


2RU form factor

NPIVNo Oversubscription

POAP

Slow Drain Automatic Recovery

Robust ArchitectureFEC

Industry wide EcoSystems

16 Gbps FC Line Rate

Affordable

Non-Disruptive Software Upgrade

Flexible Licensing

NPV

16 member PortChannel

Link Encryption

Predictable PerformanceRedundant Fan Trays

Quick ConfigurationFEC Consistent Latency

Cisco MDS + UCS Synergies

Slow Drain detection at 2.5µs

MDS 9396SSmart Zoning

Deploytoday! 96 ports


Reference


B2B credit requirement increases with

Extending optical FC SAN : B2B credit requirement

Frame Size 1 Gbps 2 Gbps 4 Gbps 8 Gbps 10 Gbps 16 Gbps

512 Bytes 2 BB/km 4 BB/km 8 BB/km 16 BB/km 24 BB/km 32 BB/km

1024 Bytes 1 BB/km 2 BB/km 4 BB/km 8 BB/km 12 BB/km 16 BB/km

2112 Bytes 0.5 BB/km 1 BB/km 2 BB/km 4 BB/km 6 BB/km 8 BB/km

Distance Speed Frame size

B2B

cre

dit

requ

irem

ent


Understanding B2B credits on MDS 9396S

• All ports in a port group can have maximum of 500 B2B credits

• Enterprise license enables extended credits that means up to 4095 B2B per port in a port group

• Best Practice: Avoid grouping all E ports in same port group/IOSlice

• Generic Formula: 0.5 B2B credits are required for every 1 KM distance at 1Gbps for standard FC frame (2112 bytes). Refer to previous slide

• CLI command: show port-resources module 1Port

Group 1

Port Group

2

Port Group

3

Port Group

4

Port Group

5

Port Group

6

Designing Scalable SAN using MDS 9396S

Technology

Transcript of Designing Scalable SAN using MDS 9396S