BRKDCT-2081 Cisco FabricPath Technology and Design (2011 London)

BRKDCT-2081

FabricPath Technology and Design

2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2081 2

Agenda

FabricPath Introduction

FabricPath Technical Overview

FabricPath and TRILL

FabricPath Use Case and Designs

FabricPath Monitoring and Troubleshooting

Summary

3 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2081

FabricPath Introduction


VLANVLAN

VLANVLAN

Access

Core

Eternal Debates on Network DesignLayer 2 or Layer 3?

Layer 3

Network

VLANVLAN

VLANVLAN

L3

L2

Simplicity (no planning/configuration required for either addressing or control

plane)

Single control plane protocol for unicast, broadcast, and multicast

Easy application development

Subnet provide fault isolation Scalable control planes with inherent provision of multi-pathing and multi-topology HA with fast convergence Additional loop-mitigation mechanism in the data plane (e.g. TTL, RPF check, etc.)

Both Layer 2 and Layer 3 are required for any network design

Cisco has solutions for both Layer 2 and Layer 3 to satisfy

Customers requirementsLayer 2?

Layer 3?


L2 Network Requirements inside DC

Maximize Bi-Sectional Bandwidth

Scalable Layer 2 domain

High Availability

Resilient control-plane

Fast convergence upon failure

Fault-domain isolation

Facilitate Application Deployment

Workload mobility, Clustering, etc.

Multi-Pathing/Multi-Topology


L2 Provides Flexibility in the Data Center

Layer 2 required by data center applications

Layer 2 is plug and play

Layer 2 is Layer 3 agnostic

With Layer 2:

Server mobility does not require interaction between Network/Server teams

Theoretically, no physical constraint on server location


L2 Requires a Tree Branches of trees never interconnect (no loop)

Spanning Tree Protocol (STP) typically used to build this tree

Tree topology implies: Wasted bandwidth increased oversubscription

Sub-optimal paths

Conservative convergence (timer-based) failure catastrophic (fails open)

11 Physical Links 5 Logical Links

S1

S2

S3


VPCdomain

Virtual Port Channel (vPC)

Introduces some changes to the data plane

Provides active/active redundancy

Does not rely on STP (STP kept as safeguard)

Limited to pair of switches (enough for most cases)

Redundancy

handled by STP

Redundancy

handled by vPC

Blocked port (STP)

Simple Building Block

Data plane based loop prevention


MAC Address Scaling & L2 Bridging

MAC addresses encode no location or network hierarchy

Default forwarding behavior in bridged network is flood

MAC filtering database limits scope of flooding

Ultimately, does not scale every switch learns every MAC

MAC Table

A

MAC Table

A

MAC Table

A

MAC Table

A

MAC Table

A

MAC Table

A

Layer 2

Domain


Network Addressing SchemeMAC v.s. IP

10.0.0.10 /24

Network Address

10.0.0.0/24

Host Address

10.0.0.10

0011.1111.1111

Non-hierarchical

Address

L2 Forwarding (Bridging)

Data-plane learning Flat address space and forwarding table (MAC everywhere!!!)

Flooding required for unknown unicastdestination

Destination MACs need to be known for all switches in the same network to

avoid flooding

0011.1111.1111 0011.1111.1111

0011.1111.1111

0011.1111.11110011.1111.1111

L3 Forwarding (Routing)

Control-plane learning Hierarchical address space and forwarding

Only forwarding to destination addresses with matching routes in the

table

Flooding is isolated within subnets No dependence on data-plane for maintaining forwarding table

10.0.0.10 20.0.0.20

10.0.0.0/24

10.0.0.0/16 20.0.0.0/16

20.0.0.0/24


The Next Era of Layer 2 NetworkWhat Can Be Improved?

Network Address Scheme: Flat Hierarchical

Additional header is required to allow L2 Routing instead of Bridging

Provide additional loop-prevention mechanism like TTL

Address Learning: Data Plane Control Plane

Eliminate the needs to program all MACs on every switches to avoid flooding

Control Plane: Distance-Vector Link-State

Improve scalability, minimize convergence time, and allow multipathing inherently

The ultimate solution needs to take both control

and data plane into consideration this time!!!


Layer 3 strengths Leverage bandwidthFast convergenceHighly scalable

Introducing Cisco FabricPathAn NX-OS Innovation for Layer 2 Networks

Simplicity Flexibility Bandwidth Availability Cost

Layer 2 strengthsSimple configurationFlexible provisioning Low cost

Resilience

Fabric

Path

"The FabricPath capability within Cisco's NX-OS offers dramatic increases in network scalability and resiliency for our service delivery data center. FabricPath extends the benefits of the Nexus 7000 in our network, allowing us to leverage a common platform, simplify operations, and reduce operational costs.

Mr. Klaus Schmid, Head of DC Network & Operating, T-Systems International GmbH


FabricPath: an Ethernet Fabric

Connect a group of switches using an arbitrary topology

With a simple CLI, aggregate them into a Fabric:

Enabling Network Fabrics

N7K(config)# interface ethernet 1/1

N7K(config-if)# switchport mode fabricpath

An open protocol based on L3 technology provides Fabric-wide intelligence and ties the elements together

FabricPath


What is a Fabric?

Externally, a Fabric looks like a single switch

Internally, a protocol adds Fabric-wide intelligence and ties the elements together. This protocol provides in a plug-and-play fashion:

Optimal, low latency connectivity any to any

High bandwidth, high resiliency

Open management and troubleshooting

Cisco FabricPath provides additional capabilities in term of scalability and L3 integration

FabricPath FabricPath


FabricPath Simplicity from the Outside

Benefits server team by providing a network Fabric that looks like a single switch Breaks down silos, permits workload mobility, provides maximum flexibility

Lowers OPEX by simplifying server team operation Reduces dependency on/interaction with network team

Web Servers App Servers New Apps

Silo 1 Silo 2 Silo 3

Web Servers

App Servers

New Apps

FabricPath Any App, Anywhere!Multi-Domain Silos

Fabric


FabricPath Simplicty from the Inside

Benefits network team by:

Reducing number of switches Higher port density

Lower oversubscription

Isolating network from the usersNo impact due to topology changes

Fabric can be upgraded/reconfigured live

Utilizing an open protocolUnicast, multicast, broadcast, VLAN pruning all controlled by single control protocol

Maintenance and troubleshooting equivalent to L3 network

Easy to extend, providing standards-compliance with Cisco value-add


FabricPath Technical Overview


Cisco Nexus Platform

Cisco NX-OS

Cisco FabricPath Overview

FabricPath encapsulation

Conversation Learning

Routing, not bridging

Built-in loop-mitigation

Time-to-Live (TTL)

RPF Check

Data Plane Innovation

Plug-n-Play Layer 2 IS-IS

Support unicast and multicast

Fast, efficient, and scalable

Equal Cost Multipathing(ECMP)

VLAN and Multicast Pruning

Control Plane Innovation

Cisco FabricPath


FabricPath versus Classic Ethernet Interfaces

STPFabricPath

Classic Ethernet (CE) Interface

Interfaces connected to existing NICs and traditional network devices

Send/receive traffic in 802.3 Ethernet frame format

Participate in STP domain Forwarding based on MAC table

FabricPath Interface

Interfaces connected to another FabricPathdevice

Send/receive traffic with FabricPath header No spanning tree!!! No MAC learning Exchange topology info through L2 ISIS

adjacency

Forwarding based on Switch ID Table

Ethernet Ethernet FabricPath Header

FabricPath interface

CE interface


FabricPath IS-IS

FabricPath IS-IS replaces STP as control-plane protocol in FabricPath network

Introduces link-state protocol with support for ECMP for Layer 2 forwarding

Exchanges reachability of Switch IDs and builds forwarding trees

Improves failure detection, network reconvergence, and high availability

Minimal IS-IS knowledge required no user configuration by default

Maintains plug-and-play nature of Layer 2

STPFabricPath

STP BPDUFabricPath IS-IS

STP BPDU


Why IS-IS?

A few key reasons:

Has no IP dependency no need for IP reachability in order to form adjacency between devices

Easily extensible Using custom TLVs, IS-IS devices can exchange information about virtually anything

Provides SPF routing Excellent topology building and reconvergence characteristics

FabricPath Port

CE Port

L2 Fabric


Basic FabricPath Data Plane Operation

Ingress FabricPath switch determines destination Switch ID and imposes FabricPath header

Destination Switch ID used to make routing decisions through FabricPath core

No MAC learning or lookups required inside core

Egress FabricPath switch removes FabricPath header and forwards to CE

STP

FabricPath Core

FabricPath interface

CE interface

STP

MAC A MAC B

S10 S20

DMACB

SMACA

Payload

DMACB

SMACA

Payload

Ingress FabricPath

Switch

Egress FabricPath

Switch

DMACB

SMACA

Payload

DSID20

SSID10

DMACB

SMACA

Payload

DSID20

SSID10

DMACB

SMACA

Payload

DMACB

SMACA

Payload

Encapsulation to creates hierarchical address scheme


Cisco FabricPath

Frame

Classical Ethernet Frame

FabricPath Encapsulation16-Byte MAC-in-MAC Header

Switch ID Unique number identifying each FabricPath switch

Sub-Switch ID Identifies devices/hosts connected via VPC+

Port ID Identifies the destination or source interface

Ftag (Forwarding tag) Unique number identifying topology and/or multidestination distribution tree

TTL Decremented at each switch hop to prevent frames looping infinitely

DMAC SMAC 802.1Q Etype CRCPayload

DMAC SMAC 802.1Q Etype PayloadCRC

(new)

FP

Tag

(32)

Outer

SA

(48)

Outer

DA

(48)

Endnode ID

(5:0)

Endnode ID

(7:6)

U/L

I/G

RS

VD

OO

O/D

L

Etype

6 bits 1 1 2 bits 1 1 12 bits 8 bits 16 bits 10 bits 6 bits16 bits

Switch IDSub

Switch IDFtag TTLPort ID

Original CE Frame


FabricPath MAC Table

Edge switches maintain both MAC address table and Switch ID table

Ingress switch uses MAC table to determine destination Switch ID

Egress switch uses MAC table (optionally) to determine output switchport

Local MACs point

to switchports

Remote MACs point

to Switch IDs

S10 S20 S30 S40

S100 S101 S200FabricPath

MAC A MAC C MAC DMAC B

FabricPath

MAC Table on S100

MAC IF/SID

A e1/1

B e1/2

C S101

D S200


S10 S20 S30 S40

S100 S200FabricPath

po1 po2 po3 po4

A B

show mac address-table dynamic

S100# sh mac address-table dynamic

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 10 0000.0000.0001 dynamic 0 F F Eth1/15

* 10 0000.0000.0002 dynamic 0 F F Eth1/15

* 10 0000.0000.0003 dynamic 0 F F Eth1/15

* 10 0000.0000.0004 dynamic 0 F F Eth1/15

* 10 0000.0000.0005 dynamic 0 F F Eth1/15

* 10 0000.0000.0006 dynamic 0 F F Eth1/15

* 10 0000.0000.0007 dynamic 0 F F Eth1/15

* 10 0000.0000.0008 dynamic 0 F F Eth1/15

* 10 0000.0000.0009 dynamic 0 F F Eth1/15

* 10 0000.0000.000a dynamic 0 F F Eth1/15

10 0000.0000.000b dynamic 0 F F 200.0.30

10 0000.0000.000c dynamic 0 F F 200.0.30

10 0000.0000.000d dynamic 0 F F 200.0.30

10 0000.0000.000e dynamic 0 F F 200.0.30

10 0000.0000.000f dynamic 0 F F 200.0.30

10 0000.0000.0010 dynamic 0 F F 200.0.30

10 0000.0000.0011 dynamic 0 F F 200.0.30

10 0000.0000.0012 dynamic 0 F F 200.0.30

10 0000.0000.0013 dynamic 0 F F 200.0.30

10 0000.0000.0014 dynamic 0 F F 200.0.30

S100#


FabricPath Control Plane Operation

FabricPath IS-IS manages Switch ID (routing) table

All FabricPath-enabled switches automatically assigned Switch ID (no user configuration required)

Algorithm computes shortest (best) paths to each Switch ID based on link metrics

Equal-cost paths supported between FabricPath switches

S10 S20 S30 S40

S100 S101 S200

FabricPath

FabricPath

Routing Table on S100

Switch IF

S10 L1

S20 L2

S30 L3

S40 L4

S101 L1, L2, L3, L4

S200 L1, L2, L3, L4

One best pathto S10 (via L1)

Four equal-cost

paths to S101

L1 L2 L4L3

Plug-n-Play L2 IS-IS manages forwarding topology


Building the FabricPath Routing Table

S10 S20 S30 S40


MAC A MAC C MAC DMAC B

L1 L2 L4L3

L5 L6 L7 L8

L9 L10 L11 L12

Switch IF

S10 L1

S20 L2

S30 L3

S40 L4

S101 L1, L2, L3, L4

S200 L1, L2, L3, L4

Switch IF

S20 L1,L5,L9

S30 L1,L5,L9

S40 L1,L5,L9

S100 L1

S101 L5

S200 L9

Switch IF

S10 L4,L8,L12

S20 L4,L8,L12

S30 L4,L8,L12

S100 L4

S101 L8

S200 L12

Switch IF

S10 L9

S20 L10

S30 L11

S40 L12

S100 L9, L10, L11, L12

S101 L9, L10, L11, L12


show fabricpath route

S100# sh fabricpath route

FabricPath Unicast Route Table

'a/b/c' denotes ftag/switch-id/subswitch-id

'[x/y]' denotes [admin distance/metric]

ftag 0 is local ftag

subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/100/0, number of next-hops: 0

via ---- , [60/0], 5 day/s 18:38:46, local


via Po1, [115/10], 0 day/s 04:15:58, isis_l2mp-default












S100#

S10 S20 S30 S40

S100 S200FabricPath

po1 po2 po3 po4

A B


When multiple forwarding paths available, path selection based on ECMP hash function

Up to 16 next-hop interfaces for each destination Switch ID

Number of next-hops installed controlled by maximum-paths command under FabricPathIS-IS process (default is 16)

Path selection based on hash function

FabricPath ECMP

S1

S100

S16


Multiple Topologies

L1

L2 L3 L4

L8L5L6 L7

L9

L10 L11 L12

L2 Fabric

Topology: A group of links in the Fabric.

By default, all the links are part of topology 0.

Other topologies can be created by assigning a subset of the links to them.

A link can belong to several topologies

A VLAN is mapped to a unique topology

Topologies can be used for traffic engineering, security etc

Topology 0

Topology 1

Topology 2


Conversational MAC Learning

MAC learning method designed to conserve MAC table entries on FabricPath edge switches

FabricPath core switches do not learn MACs at all

Each forwarding engine distinguishes between two types of MAC entry:

Local MAC MAC of host directly connected to forwarding engine

Remote MAC MAC of host connected to another forwarding engine or switch

Forwarding engine learns remote MAC only if bidirectional conversation occurring between local and remote MAC

MAC learning not triggered by flood frames

Conversational learning enabled in all FabricPath VLANs


MAC C


FabricPath Core

MAC A

MAC B

FabricPath

MAC Table on S100

MAC IF/SID

A e1/1 (local)

B S200 (remote)

S100

S200

S300

FabricPath

MAC Table on S200

MAC IF/SID

A S100 (remote)

B e12/1(local)

C S300 (remote)

FabricPath

MAC Table on S300

MAC IF/SID

B S200 (remote)

C e7/10 (local)



500

MACs

500

MACs

500

MACs

500

MACs

250

MACs

250

MACs

250

MACs

250

MACs

ALL MACs needs to be learn on EVERY Switch

Large L2 domain and virtualization present challenges to MAC Table scalability

STP Domain

Local MAC: Source-MAC Learning only happen to traffic received on CE Ports

Remote MAC: Source-MAC for traffic received on FabricPath Ports are only learned if Destination-MAC is already known as Local

S11

A C

B

L2 Fabric

MAC IF

C 3/1

A S11

MAC IF

B 2/1

MAC IF

Optimize Resource Utilization Learning only the MAC addresses required


FabricPath TreeUsed for forwarding L2 multi-destination traffic (Unknown

Unicast, Broadcast, and Multicast) inside the L2 Fabric

Tree topology is required to forward multi-destination traffic properly

One Ingress Switch Many Egress Switches

Same method is also used by L3 (e.g. PIM Source Tree/Shared Tree)

One or more Root devices are first elected for the L2 Fabric

A Tree spanning from each Root is then formed and a network-wide unique ID is assigned to it

Support for multiple Trees allows Cisco FabricPath to support multipathing even for multi-destination traffic

Ingress Switch determines the Tree for each traffic flow

S100 S105

S200

S101

A

L2 Fabric

CFabricPath Port

CE Port

S100 S200

S1 S2 S16

L1 L2

L16

L101 L102L116

Root for

Tree #1

Tree # IF

1 L1, L101


FabricPath Multidestination Trees

Multidestination traffic constrained to loop-free trees touching all FabricPath switches

Root switch assigned for each multidestination tree in FabricPath domain

Loop-free tree built from each Root and assigned a network-wide identifier (Ftag)

Support for multiple multidestination trees provides multipathing for multi-destination traffic

Two trees supported in NX-OS release 5.1

S10 S20 S30 S40


Root for

Tree 1

S10

S100

S101

S200

S20

S30

S40

Logical

Tree 1

Root for

Tree 2

S40

S100

S101

S200

S10

S20

S30

Logical

Tree 2

Root Root


S10 S20 S30 S40


Root for

Tree 1

Root for

Tree 2

Multidestination Trees and Role of the Ingress FabricPath Switch

Ingress FabricPath switch determines which tree to use for each flow

Other FabricPath switches forward based on tree selected by ingress switch

Broadcast and unknown unicast typically use first tree

Hash-based tree selection for multicast, with several configurable hash options

Multidestination

Trees on Switch 100

Tree IF

1 L1,L2,L3,L4

2 L4

L1 L2 L4L3

L5 L6 L7 L8

L9 L10 L11 L12


Putting It All Together Host A to Host B(1) Broadcast ARP Request

S10 S20 S30 S40


Root for

Tree 1

Root for

Tree 2

MAC A MAC B

Multidestination

Trees on Switch 100

Tree IF

1 L1,L2,L3,L4

2 L4

DMACFF

SMACA

Payload

DSIDFFFtag1

SSID100

Broadcast

DMACFF

SMACA

Payload

Multidestination

Trees on Switch 10

Tree IF

1 L1,L5,L9

2 L9

L1 L2 L4L3

L5 L6 L7 L8

L9 L10 L11 L12

Ftag

Ftag

DMACFF

SMACA

Payload

DSIDFFFtag1

SSID100

FabricPath

MAC Table on S200

MAC IF/SID

Multidestination

Trees on Switch 200

Tree IF

1 L9

2 L9,L10,L11,L12

FabricPath

MAC Table on S100

MAC IF/SIDMAC IF/SID

A e1/1 (local)

DMACFF

SMACA

Payload

Learn MACs of directly-connected

devices unconditionally

Dont learn MACs in flood frames


Putting It All Together Host A to Host B(2) Unicast ARP Reply

S10 S20 S30 S40


MAC A MAC B

Multidestination

Trees on Switch 100

Tree IF

1 L1,L2,L3,L4

2 L4

DMACA

SMACB

Payload

DSIDMC1Ftag1

SSID200

Ftag

DMACA

SMACB

Payload

Multidestination

Trees on Switch 10

Tree IF

1 L1,L5,L9

2 L9

Ftag

Unknown

DMACA

SMACB

Payload

DSIDMC1Ftag1

SSID200

FabricPath

MAC Table on S200

MAC IF/SID

Multidestination

Trees on Switch 200

Tree IF

1 L9

2 L9,L10,L11,L12

FabricPath

MAC Table on S100

MAC IF/SID

A e1/1 (local)DMACA

SMACB

Payload

MAC IF/SID

B e12/2 (local)

A

MAC IF/SID

A e1/1 (local)

B S200 (remote)

L1 L2 L4L3

L5 L6 L7 L8

L9 L10 L11 L12

A If DMAC is known, then learn remote MAC


FabricPath

MAC Table on S200

MAC IF/SID

B e12/2 (local)

FabricPath

MAC Table on S100

MAC IF/SID

A e1/1 (local)

B S200 (remote)

Putting It All Together Host A to Host B(3) Unicast Data

S10 S20 S30 S40


MAC A MAC BS200

DMACB

SMACA

Payload

L1 L2 L4L3

L5 L6 L7 L8

L9 L10 L11 L12

S200

DMACB

SMACA

Payload

DSID200Ftag1

SSID100

MAC IF/SID

A S100 (remote)

B e12/2 (local)

DMACB

SMACA

Payload

B B

FabricPath Routing

Table on S100

Switch IF

S10 L1

S20 L2

S30 L3

S40 L4

S101 L1, L2, L3, L4

S200 L1, L2, L3, L4

DMACB

SMACA

Payload

DSID200Ftag1

SSID100

FabricPath Routing

Table on S30

Switch IF

S200 L11

FabricPath Routing

Table on S30

Switch IF

S200 S200

Hash


Loop Mitigation with FabricPath

Minimize impact of transient loop with TTL and RPF Check

STP Domain

Block redundant paths to ensure loop-free topology

Frames loop indefinitely if STP failed

Could results in complete network melt-down as the result of flooding

Root

L2 Fabric

S1

S10

S2

TTL=3

TTL=2 TTL=1

TTL=0

TTL is part of FabricPath header

Decrement by 1 at each hop

Frames are discarded whenTTL=0

RPF check for multicast based on tree info

Root


VLAN Pruning in L2 Fabric

VL

10

VL

20

VL

30

VL

10

VL

30

VL

20

L2 FabricShared

Broadcast Tree

L2 Fabric

VLAN 10

L2 Fabric

VLAN 20

L2 Fabric

VLAN 30

Switches indicate locally interested VLANs to the rest of the L2 Fabric

Broadcast traffic for any VLAN only sent to switches that have requested for it


STP Interaction

L2 Fabric is presented as a single bridge to all connected CE devices

L2 Fabric should be the root for all connected STP domains. CE ports will be put into blocking state when better BPDU is received (rootguard)

No BPDUs are forwarded across the fabric (terminated on CE ports)

L2 Fabric

Classical Ethernet

(STP)

FabricPath(L2 IS-IS)

STP Domain 1

STP Domain 2

FabricPath Port

CE Port


vPC Enhancement for FabricPath

For Switches at L2 Fabric Edge

vPC is still required to provide active/active L2 paths for dual-homed CE devices or clouds

However, MAC Table only allows 1-to-1 mapping between MAC and Switch ID

Each vPC domain is represented by an unique Virtual Switch to the rest of L2 Fabric

Switch ID for such Virtual Switch is then used as Source in FabricPath encapsulation

L2 Fabric

S1 S2

A

B

S3

MAC Table

A ???

MAC Table

B S3

B A Payload

B A PayloadS2S3B A PayloadS1S3

MAC Table

A S4

vPC

L2 Fabric

S1 S2

B

S3

B A Payload

A

S4

B A PayloadS4S3 B A PayloadS4S3

vPC+MAC Table

B S3


Connect L3 or Services to L2 Fabric

Layer 3 Network

L3

L2 FHRP

FHRPActive

Mu

lti-

path

ing

FabricPath enables multipathingfor bridged traffic

However, FHRP allows only 1 active gateway for each host, therefore prevent traffic that needs to be routed to take advantage of multi-pathing

Provide active/active data-plane for FabricPath with no change to existing FHRP

Allow multi-pathing even for routed traffic

Same feature can be leveraged by service nodes as well

L2 Fabric

VMAC

Layer 3 Network

L3

L2 FHRP

FHRPActive

Mu

lti-

path

ing

L2 Fabric

VMAC VMAC

vPC+


VPC+

VPC+ allows dual-homed connections from edge ports into FabricPath domain with active/active forwarding

CE switch, Layer 3 router, dual-homed server, etc.

VPC+ requires F1 modules with FabricPath enabled in the VDC

Peer-link and all VPC+ connections must be to F1 ports

VPC+ creates virtual FabricPath switch for each VPC+-attached device to allow load-balancing within FabricPath domain

F1F1

VPC+F1

F1F1

S1 S2

po3

F1

F1F1

VPC+F1

F1F1

S1 S2

po3

F1

Host AS4L1,L2S3

Host A

Host A

L1 L2

S3

L1 L2

S4

Physical

Logical

Virtual Switch 4 becomes next-hopfor Host A in FabricPath domain

FabricPath

CE


MAC A

VPC+ Physical Topology

S10 S20 S30 S40

S100 S200FabricPath

MAC B MAC C

Peer link and

PKA required

Peer link runs as

FabricPath core port

VPCs configured

as normal

No requirements for

attached devices other

than channel support

VLANs must be

FabricPath VLANs


VPC+ Logical Topology

MAC A

S10 S20 S30 S40

S100 S200FabricPath

MAC B MAC C

S1000

Virtual switch

introduced


Remote MAC Entries for VPC+

MAC A

S10 S20 S30 S40

S100 S200FabricPath

MAC B MAC C

S1000


Legend:




---------+-----------------+--------+---------+------+----+------------------

* 10 0000.0000.000c dynamic 1500 F F Eth1/30

10 0000.0000.000a dynamic 1500 F F 1000.11.4513

S200#

po1po2

1/30


FabricPath Routing for VPC+

MAC A

S10 S20 S30 S40

S100 S200FabricPath

MAC B MAC C

S1000

S200# sh fabricpath route topology 0 switchid 1000










S200#

po1po2

1/30


SVI SVI

VPC+ and Active/Active HSRP

With VPC+ and SVIs in mixed-chassis, HSRP Hellos sent with VPC+ virtual switch ID

FabricPath edge switches learn HSRP MAC as reached through virtual switch

Traffic destined to HSRP MAC can leverage ECMP if available

Either VPC+ peer can route traffic destined to HSRP MAC

HSRP Active HSRP Standby

MAC A

S10 S20 S30 S40

S100 S200FabricPath

MAC B MAC C

S1000

po1po2

1/30

DMAC0002

SMACHSRP

Payload

DSIDMC

SSID1000


HSRP MAC on Edge Switches

SVI SVI

HSRP Active HSRP Standby

MAC A

S10 S20 S30 S40

S100 S200FabricPath

MAC B MAC C

S1000

po1po2

S200# sh mac address-table dynamic address 0000.0c07.ac0a

Legend:




---------+-----------------+--------+---------+------+----+------------------

10 0000.0c07.ac0a dynamic 0 F F 1000.0.1054

S200#


Edge Devices Integration

Hosts see a single default gateway

The fabric provide them transparently with multiple simultaneously active default gateways

Allows extending the multipathing from the inside to the fabric to the L3 domain outside the fabric

Hosts can leverage multiple L3 default gateways

FabricPath

A

s3

dgdg

L3

dg


Layer 3 Integration

The fabric provides seamless L3 integration

An arbitrary number of routed interfaces can be created at the edge or within the fabric

Attached L3 devices can peer with those interfaces

The hardware is capable of handling million of routes

SVIs anywhere

FabricPathL3

L3


L3

Alternatives for N-Way Layer 3 EgressVLAN Splitting with Active/Active HSRP in VPC+

S1 S4

L1

FabricPath

CE

S3S2

L2

L4

VLANs x: GWY MAC XL1, L2VLANs y: GWY MAC YL3, L4

VPC+VPC+

HSRP HSRPActive/Active HSRP

for VLANs X

GWY MAC X

L3

Leverages benefit of VPC+ active/active HSRP

Each router still has interface in all VLANs but not running HSRP

Does require PL/PKA, and mixed chassis

Active/Active HSRP

for VLANs Y

GWY MAC Y


FabricPath Configuration

No L2 IS-IS configuration required

New feature-set keyword allows multiple conditional services required by FabricPath (e.g. L2 IS-IS, LLDP, etc.) to be enabled in one shot

Simplified operational model only 3 CLIs to get FabricPathup and running

L2 Fabric

FabricPath Port

CE Port

N7K(config)# feature-set fabricpath

N7K(config)# vlan 10-19

N7K(config-vlan)# mode fabricpath

N7K(config)# interface port-channel 1



FabricPath comparison

Transparent

Bridging

vPC FabricPath IP Routing

Control Protocol Spanning

Tree

Spanning

Tree

IS-IS IS-IS/ EIGRP/

OSPF etc

Default forwarding behavior Flood Flood Drop Drop

Data plane loop protection None None RPFC, TTL RPFC, TTL

Frames/packets forwarded

along the shortest pathNo Yes

(limited topologies)

Yes Yes

Multiple paths between

nodesNo Yes

(limited topologies)

Yes, ECMP Yes, ECMP

Transparent to IP and other

L3 protocolsYes Yes No

Configuration less

addressingYes Yes No


Cisco FabricPath Feature SetValue-Add Enhancements

16-Way Equal Cost Multipathing(ECMP) at Layer 2

FabricPath HeaderHierarchical Addressing with built in loop mitigation (RPF,TTL)

Conversational MAC LearningEfficient use of hardware resource by learning only MACs for interested hosts

Interoperability with existing classic Ethernet networks

VPC + allows VPC into a L2 Fabric

STP Boundary Termination

Multi-Topology providing traffic engineering capabilities

Cisco FabricPath

Up to

16Way L2

ECMP

Up to 16-Way

L2 ECMP

Cisco FabricPath


FabricPath & TRILL


TRILL Standardizing Multi-pathing

IETF RFC 5556 defines Transparent Interconnection of Lots of Links (TRILL)

TRILL is a standards based implementation of Layer 2 Multi-pathing

Lot of similarities between Ciscos current implementation and TRILL

TRILL HW Frame format finalized

Final control plane (SW implementation) to be standardized by end of the year

IETF standard for Layer 2 multipathing

Driven by multiple vendors, including Cisco

Base protocol RFC ready for standardization but waiting on dependent standards

Control-plane protocol RFCs still in process

Target for standard completion is early CY2011

http://datatracker.ietf.org/wg/trill/


What Is the Relationship between FabricPath and TRILL?

a set of Layer 2 multipathing technologies

FabricPath initial release runs in a Native mode that is Cisco-specific, using proprietary encapsulation and control-plane elements

Nexus 7000 F1 I/O modules and Nexus 5500 HW are capable of running both FabricPath and TRILL modes


FabricPath & TRILL Feature Summary

FS-link is a superset of TRILL

L2MP TRILL

Frame routing (ECMP, TTL, RPFC etc)

Yes Yes

vPC+ Yes No

FHRP active/active Yes No

Multiple topologies Yes No

Conversational learning Yes No

Inter-switch links Point-to-point only Point-to-point OR shared

Base protocol specification is now a proposed IETF standard (March 2010)

Control plane specification will become a proposed standard within months


Examples of FabricPath Use Case


FabricPath Design Guidance

Industry has converged on a handful of well-understood designs/network topologies

Largely driven by constraints of STP, and density limits of switches

Designs will necessarily evolve

Not only what can/cannot be built today versus in future, but how people think about L2 designs in general


Scaling Bandwidth with FabricPathExample: 2,048 X 10GE Server Design

16X improvement in bandwidth performance

From 74 managed devices to 12 devices

2X+ increase in network availability

Simplified IT operations

Traditional Spanning Tree Based Network FabricPath Based Network

Fu

lly N

on

-Blo

ckin

g

2, 048 Servers

8 Access Switches

Network Fabric

64 Access Switches

2, 048 Servers

Blocked Links

Ov

ers

ub

scri

pti

on

16:1

8:1

2:1

4

Pods


32 Chassis

16 Chassis

16-way ECMP

8,192 10GE ports

512 10GE FabricPath ports per system

256 10GE FabricPath Ports

160 Tbps System Bandwidth

Open I/O Slots for

connectivity

Spine Switch

Edge Switch

16-port Etherchannel

FabricPath

HPC Requirements

HPC Clusters require high-density of compute nodes

Minimal over-subscription

Low server to server latency

FabricPath Benefits for HPC

FabricPath enables building a high-density fat-tree network

Fully non-blocking with FabricPath ECMP & port-channels

Minimize switch hops to reduce server to server latencies

Use Case: High Performance ComputeBuilding Large Scalable Compute Clusters


Workload Flexibility with FabricPathExample: Removing Data Center Silos

Single domain

Pooled compute resources

Increased agility

Seamless data center wideworkload mobility

Responsive

Virtualized Applications movewithin minutes vs. days

Capex and Opex savings

Maximize resource utilization, simplify IT operations

Web Servers App Servers New Apps

Silo 1 Silo 2 Silo 3

Web Servers

App Servers

New Apps

Single Domain Any App, Any where!

Network Fabric

Multi-Domain Silod


Use Case: L2 Internet Exchange Point

IXP Requirements

Layer 2 Peering enables multiple providers to peer their internet routers with one another

10GE non-blocking fabric

Scale to thousands of ports

FabricPath Benefits for IXP

Transparent Layer 2 fabric , No STP at core, simple to manage

Scalable to thousands of ports

Bandwidth not limited by chassis / port-channel limitations

N+1 redundancy in distribution

Large bisectional bandwidth at distribution

Provider A Provider B

Provider C Provider D


L3

Classical POD with FabricPath

FabricPath vs. vPC/STP

FabricPath POD

Simple configuration (no peer link, no pair of switches, no port channels)

Total flexibility in design and cabling

Seamless L3 integration

No STP, no traditional bridging (no topology changes, no sync to worry about, no risk of loops)

Scale mac address tables with conversational learning

Unlimited bandwidth, even if hosts are single attached

Can extend easily and without operational impact

vPC POD

L3 Core


L3

FabricPath Core

Efficient POD Interconnect

vPC+ PODvPC+ POD

FabricPath in the Core

VLANs can terminate at the distribution or extend between PODs.

STP is not extended between PODs, remote PODs or even remote data centers can be aggregated.

Bandwidth or scale can be introduced in a non-disruptive way

L2+L3

FabricPath

Core


Combining FabricPath PODs and Core

Allows Tier Consolidation

3

2

L3

1L2+L3

FabricPath

2

3

L3

FabricPath

3

1

L3

FabricPath


FabricPath at the Edge

E

1/10G connectivity to Nexus 7000

1/10G connectivity to Fabric Extender attached

to Nexus 7000

1/10G connectivity to Nexus 5500

1/10G connectivity to Fabric Extender attached

to Nexus 5500

A B

C

D

E

A B C D


Migration of Existing Designs

Emphasis on preserving existing topologies without major disruption

Evolution rather than revolution in existing DC network

Assumes DC isnt pure Nexus

Phases:

Integrate Nexus 7000 with F1 modules into existing Aggregation

Migrate to VPC+

Migrate Access devices to FabricPath

Interconnect FabricPath Pods

Pod scale-out


Migration Phases

Only the core of the network needs to be running L2MP

Simple Integration of Classical Ethernet

vPC+

FabricPath

7K access 7K or 5K access + FEX

Cairo (maint)Cairo End CY2010

CE access

Radar


L3

Fabric Module Integration

L3

CE

Pod 1VLANs 100-199

Pod 2VLANs 200-299

Pod 3VLANs 300-399

Active/Active HSRPfor VLANs 100-199



VPC VPC VPC

Motivations: minimize STP, use high-density, low-cost F1 modules at aggregation layer

Understand East-West capacity requirements (160Gproxy L3 per agg switch in 5.1)

North-South bandwidth already limited by uplink capacity

160G proxy

L3 per switch

Peer link runs in

CE mode Downlinks

on F1 modules

Uplinks on M1

modules

Adding F1 modules to agg (either as part of Catalyst 6500 to Nexus 7000 migration or adding F1 cards into agg that already has M1 modules)

Uplinks are on M1 modules (L3 links to core)

Downlinks on F1 modules (L2 agg to access)

Uses standard VPC with peer link in CE mode, providing active/active HSRP forwarding at agg layer

Access could be anything 7k, 6k, 5k, 5k+FEX, or any other box


L3

L3

CE

Pod 1VLANs 100-199

Pod 2VLANs 200-299

Pod 3VLANs 300-399




VPC+ VPC+ VPC

VPC+ in Localized Pods Motivations: prepare for scale-out and

VLAN anywhere while preserving investment in STP devices

Note that change from VPC to VPC+ is disruptive

CE

Peer link runs in

FabricPath mode

Only change here is migration from VPC to VPC+, in preparation to add FabricPath devices in access combined with VPC+ attached legacy CE devices


L3

L3

Pod 1VLANs 100-199

Pod 2VLANs 200-299

Pod 3VLANs 300-399




VPC+ VPC+ VPC

Migrating to FabricPath Pods

Motivations: prepare for scale-out and VLAN anywhere

FabricPath

Pod 1VLANs 100-199

Keep VPC+ for

active/active

forwarding

Migrate all or part of each pod to FabricPath

Keep VPC+ to provide active/active HSRP

FabricPath here

assumes Nexus 5500

Leverage VPC+ for

existing Nexus 5000


L3

Meshed Aggregation Layer

L3

FabricPath

Pod 1VLANs 100-299

Pod 2VLANs 100-299

Pod 3VLANs 300-399


VPC

Motivations: Consolidation; VLAN anywhere with FabricPath network

Number of Pods you can combine limited by abilty to fully mesh aggregation switches

Reduced cabling burden vs direct access connect, but has gateway and scale limits

VPC+ VPC+



Affinity for 100-199 Affinity for 200-299

Backbone/mesh agg layer connections provide VLAN anywhere capability among connected FabricPath Pods

Still have Layer 3 VLAN affinity at Pod level HSRP for particular VLAN only lives in one Pod


L3

Parallel FabricPath Core

L3

FabricPath

Pod 1VLANs 100-299

Pod 2VLANs 100-299



VPC+ VPC+

FabricPath Core

Pod 3VLANs 300-399


VPC


Motivations: Consolidation and whole-network scale

Removes access connections and aggregation mesh limitations Meshed agg model overly complex

after a certain point

Add FabricPath core parallel to L3 core to interconnect FabricPath Pods


L3 L3

Parallel FabricPath Core with VDCs

L3

FabricPath

Pod 1VLANs 100-299

Pod 2VLANs 100-299



VPC+ VPC+

FabricPathCore VDC

FabricPathCore VDC

Layer 3Core VDC Layer 3

Core VDC

Pod 3VLANs 300-399


VPC


Exact same model as prior slide but with VDCs instead of separate physical switches


L3

Pod Build-Out with ParallelFabricPath Core

L3

FabricPath

Pod 1VLANs 100-299

Pod 2VLANs 100-299

FabricPath Core

Pod 3VLANs 300-399


VPC

N-Way Active FHRPfor VLANs 100-299

Motivations: Consolidation and per-Pod scale

Requires n-way FHRP Add additional capacity in each

Pod using more agg switches

Not all aggs have to connect to FabricPath or L3 core necessarily


L3

SVI SVI

Standby

SVISVI

SVI SVI

L3 Egress 3 L3 Egress 4L3 Egress 1 L3 Egress 2

FabricPath Core with L3 Access

OSPF etc.

S1 S4

FabricPath

CE

S3S2

VPC+VPC+ VPC+

HSRP

ActiveStandby

OSPF etc.

Active

HSRP HSRP

OSPF

Scales L3 at the edge

Can extend VLANs through FabricPathbackbone (no hard requirement to terminate L3 at edge VPC+ peers)

VLANs still have affinity to L3 access pair

Can extend some

or all VLANs into

FabricPath core

Requires FabricPath

and L3 support on 5500


L3

SVI SVI

Standby

SVISVI

SVI SVI

L3 Egress 3

L3 Egress 1

FabricPath Core with L3 Access

OSPF etc.

S1 S4

FabricPath

CE

S3S2

VPC+VPC+ VPC+

HSRP

ActiveStandby

OSPF etc.

Active

HSRP HSRP

OSPF

Scales L3 at the edge

Can extend VLANs through FabricPath backbone (no hard requirement to terminate L3 at edge VPC+ peers)

VLANs still have affinity to L3access pair

FP extended to core

Can extend some

or all VLANs into

FabricPath core

Requires FabricPath

and L3 support on 5500

SVI SVI


Monitoring and Troubleshooting FabricPath


Troubleshooting FabricPath

Leverage the same tooling for L3 technologies

Routing table

Link-state database

Distribution trees

ECMP path selection

Pong L2 Ping + Traceroute

Provide info on all devices on a given path in L2 Fabric

Check on link health

Performance Profiling across FabricPath

Through IEEE 1588 timestamp and pong to help estimate average end-to-end latency

Improved Visibility for Layer 2 Evolution


S10 S20 S30 S40

S100 S200FabricPath

po1 po2 po3 po4

A B

show mac address-table dynamic


Legend:




---------+-----------------+--------+---------+------+----+------------------

* 10 0000.0000.0001 dynamic 0 F F Eth1/15

* 10 0000.0000.0002 dynamic 0 F F Eth1/15

* 10 0000.0000.0003 dynamic 0 F F Eth1/15

* 10 0000.0000.0004 dynamic 0 F F Eth1/15

* 10 0000.0000.0005 dynamic 0 F F Eth1/15

* 10 0000.0000.0006 dynamic 0 F F Eth1/15

* 10 0000.0000.0007 dynamic 0 F F Eth1/15

* 10 0000.0000.0008 dynamic 0 F F Eth1/15

* 10 0000.0000.0009 dynamic 0 F F Eth1/15

* 10 0000.0000.000a dynamic 0 F F Eth1/15

10 0000.0000.000b dynamic 0 F F 200.0.30

10 0000.0000.000c dynamic 0 F F 200.0.30

10 0000.0000.000d dynamic 0 F F 200.0.30

10 0000.0000.000e dynamic 0 F F 200.0.30

10 0000.0000.000f dynamic 0 F F 200.0.30

10 0000.0000.0010 dynamic 0 F F 200.0.30

10 0000.0000.0011 dynamic 0 F F 200.0.30

10 0000.0000.0012 dynamic 0 F F 200.0.30

10 0000.0000.0013 dynamic 0 F F 200.0.30

10 0000.0000.0014 dynamic 0 F F 200.0.30

S100#

Local mac


show fabricpath route

S10 S20 S30 S40

S100 S200FabricPath

po1 po2 po3 po4

A B

Topology ID: 0

Switch ID: 100

Subswitch ID:0 used for vPC+

S100# sh fabricpath route








via ---- , [60/0], 5 day/s 18:38:46, local














S100#


Abstracted Fabric View

Identify fabric hot-spots

FabricPath state awareness

Traffic Monitoring

Frames distribution visibility

Threshold crossing alerts for bandwidth management

Troubleshooting

Visualize unicast, multicast and broadcast paths

Check reachability between source and destination nodes

Configuration Expert

Manage FabricPath topologies with Wizard tools

Simplify fine-tuning FabricPath

Up

to

16-W

ay L

2 E

CM

P

Cisco FabricPath

Classical Ethernet Classical Ethernet

FabricPath: In Control with DCNM


Summary


N7K(config)# feature-set fabricpath

N7K(config)# fabricpath switch-id

N7K(config)# interface ethernet 1/1


FabricPath is Simple

No L2 IS-IS configuration required

Single control protocol for unicast, multicast, vlan pruning

L2 Fabric

FabricPath Port

CE Port

1/1


FabricPath is Efficient & ResilientShortest path, Multi-Pathing, High-availability

A

L1

L2

S1 S2 S3 S4

S11 S12 S42

L2 Fabric

L3

L4

B

Shortest path for low latency Up to 256 links active between any 2 nodes Multipathing over all links increase availability High availability with N+1 path redundancy Enhanced redundancy models No STP - Fast convergence

FabricPath

Routing Table

Switc

h

IF

S42 L1, L2, L3, L4


FabricPath is Scalable

Safe Data Plane, Conversational learning

TTL and RFP check the data plane protect against loops L2 can be extended in the data center (while STP is segmented)

Conversational learning allows scaling mac address tables at the edge

Classical Ethernet

Mac Address Table

A

S11 S42

FabricPath (no mac address learning in the Fabric)

B

MAC IF

A 1/1

B S42

Classical

Ethernet

Classical Ethernet

Mac Address Table

Classical Ethernet

Mac Address Table

MAC IF

MAC IF

A S11

B 1/1

S22


Key Takeaways

Fabric Path enables network fabric scalability, flexibility, availability and resiliency

Innovations in FabricPath will change long-standing Layer 2 networking design paradigms

FabricPath will evolve going forward

Hardware, software, and design options will only increase our flexibility and scale

Nexus hardware available has FabricPath and TRILL capability


Questions?


Breakout Sessions of Interest

BOFDCT-1503: Design Considerations to Constructing Low Latency and High Performance Architectures

BRKDCT-2079: The Evolution of Data Center Networks

BRKDCT-2951: Deploying Nexus 700 in Data Center Networks

BRKDCT-2080: Massivley Scalable Data Center Architectures

BRKDCT-2399: Technologies Transforming the Data Center

TECDCT-2781: Deployment Considerations for Interconnecting Distributed Virtual Data Centers

TECVIR-2002: Enabling the Cloud: Data Center Virtualization-Applications, Compute, Networking and Best Practices

PNLDCT-6884: 2010: the Year of the 40 Gig And 100 Gig Ethernet Standard


Source: Cisco Press

BRKDCT-2081 Recommended Reading


We value your feedback - don't forget to complete your online session evaluations after each session. Complete 4 session evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Networkers 20th Anniversary t-shirt.

All surveys can be found on our onsite portal and mobile website: www.ciscoliveeurope.com/connect/mobi/login.ww

You can also access our mobile site and complete your evaluation from your mobile phone:

1. Scan the Access Code(See http://tinyurl.com/qrmelist for software,

alternatively type in the access URL)

2. Login

3. Complete and Submit the evaluation

Please complete your Session Survey


Cisco FabricPath Technology and Design

Hour Technical Level: Intermediate

Abstract: The session provides a practical approach to Cisco's implementation of FabricPath technology to enable scalable, simplified L2 low latency high performance switching fabrics. Technology, Implementation details, use case/deployment considerations, synergy and interaction with classic Ethernet environments will be explored in addition to a look at the draft IETF TRILL (Transparent Interconnect of Lots of Links) standard.

BRKDCT-2081 Cisco FabricPath Technology and Design (2011 London)

Documents

Transcript of BRKDCT-2081 Cisco FabricPath Technology and Design (2011 London)