VSS VS VPC.pdf

8/20/2019 VSS VS VPC.pdf

1/124

BRKCRS-1930

VPC & VSS: Operation and Troubleshooting


2/124

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKCRS-1930 2

VSS and VPC

No blocked ports, More usable bandwidth, Load-sharing

Distribution or link failure != network reconvergence

…enable us to build EtherChannel to 2 separate

switches and transform network building blockto thisfrom this …or, logically


3/124


Goals

Understand generalconcepts of VPC on Nexus7000 and VSS on Catalyst6500

Study the impact of VPC andVSS on bridging and routing

Learn how to troubleshootVPC and VSS


4/124


Spirit of this session

Simple description on how things work

Special cases

Troubleshooting

More on the topic

Cisco Catalyst Virtual Switching System(BRKCRS-3468)

Advanced Enterprise Campus Design: Virtual Switching System(BRKCRS-3035)

Deploying Virtual Port Channel in NXOS(BRKDCT-2048)


5/124

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKCRS-1930

VSS


6/124


VSS Agenda

Initialization

Internal redundancy considerations

Spanning Tree

1st hop redundancy

Traffic forwarding

Multicast considerations


7/124© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKCRS-1930 7

VSS

1 active redundant control plane

single config single point of management

2 active data planes

Standby switch is essentially a

set of additional linecards

Control messages and Data

frames flow between active and

standby via VSL(can be seen as backplane

extension)

Special encapsulation on VSL

frames to carry additional

information

ActiveData Plane

ActiveControl Plane

ActiveData Plane

StandbyControl Plane

MEC

VSL

Dual-Active

detection link

Active Standby

VSS domain



VSS initializationBefore the Virtual Switch domain can become active, the Virtual Switch Link

(VSL) must be brought online to determine Active and Standby roles. Theinitialization process essentially consists of 3 steps:

Role Resolution Protocol (RRP) used to determine compatible Hardware and

Software versions to form the VSL as well as determine which switch becomes

Active and Hot Standby from a control plane perspective

LMP LMP

RRPRRP

Link Management Protocol (LMP) used to track and reject Unidirectional Links,

Exchange Chassis ID and other information between the 2 switches

Link Bringup to establish connectivity with remote chassis1

2

3



Troubleshooting VSS: quick sanity check

vss# sh switch virtualSwitch mode : Virtual Switch

Virtual switch domain number : 111Local switch number : 1Local switch operational role: Virtual Switch ActivePeer switch number : 2

vss# sh switch virtual linkVSL Status : UPVSL Uptime : 18 hours, 38 minutesVSL SCP Ping : PassVSL ICC Ping : PassVSL Control Link : Te1/6/1

vss# sh switch virtual link portLMP summaryLink info: Configured: 2 Operational: 1

Peer Peer Peer Peer Timer(s)runningInterface Flag State Flag MAC Switch Interface (Time remaining)

--------------------------------------------------------------------------------Te1/5/4 v link_down - - - -Te1/6/1 vfs operational vfs 0007.0d72.4800 2 Te2/6/1 T4(960ms)

T5(29.98s)...vss# sh redundancy states

my state = 13 -ACTIVEpeer state = 4 -STANDBY COLD

Mode = Duplex

...

In VSS mode? Domain# unique for each VSS?

Role of this switch

Peer-switch visible?

VSL is up?

Link used to carry control plane

messages (ICC, IPC, SCP) VSL member-links state

Redundancy mode SSO?



Troubleshooting VSL:counters

vss# sh switch virtual link counters

Port InOctets InUcastPkts InMcastPkts InBcastPktsPo10 3084500343 31059 7382085 1046088Te1/6/4 523470151 139662 1323349 1045940Te1/6/5 2814244020 11346 6883221 258

Port OutOctets OutUcastPkts OutMcastPkts OutBcastPktsPo10 1457635126 1467466 9890548 0Te1/6/4 363835687 264788 2732502 0Te1/6/5 1214900160 1202788 8103037 0...

Port Align-Err FCS-Err Xmit-Err ...Po10 0 0 0 ...

Te1/6/4 0 0 0 ...Te1/6/5 0 0 0 ...Port Single-Col Multi-Col Late-Col ...Po10 0 0 0 ...Te1/6/4 0 0 0 ...Te1/6/5 0 0 0 ...Port SQETest-Err Deferred-Tx IntMacTx-Err ...Po10 0 0 0 ...Te1/6/4 0 0 0 ...Te1/6/5 0 0 0 ...

Aside from packet/bit rate this is

one-stop-shop command for VSL

packet and error counters

Always take 2-3 samples

All errors should be at or near zero

and most importantly not

incrementing (giants are ok)



Troubleshooting VSL: LMPvss# sh switch virtual link detail...LMP summary

...

LMP neighbors

Peer Group info: # Groups: 1 (* => Preferred PG)

PG # MAC Switch Ctrl Interface Interfaces---------------------------------------------------------------*1 0004.9bbe.ac00 2 Te1/6/4 Te1/6/4, Te1/6/5...LMP hello timer

...LMP FSM info

sm(vslp_lmp 6/4), running yes, state operationalLast transition recorded: (hello)-> operational (t4_exp)-> operational (hello)->operational (hello)-> operational (t4_exp)-> operational (hello)-> operational...LMP counters

Tx RxInterface OK Fail Bidir Uni Fail Bad--------------------------------------------------------------------Te1/6/4 805969 0 806270 7 0 0Te1/6/5 640674 0 640726 3 0 0

Rx error detailsInterface My info My info Bad MAC Bad switch Domain id Peer info

mismatch absent Address id mismatch mismatch-------------------------------------------------------------------------------

Te1/6/4 0 7 0 0 0 0Te1/6/5 0 3 0 0 0 0

Complete information about LMP

layer of VSLP

At least 1 link should be operational

Should see a neighbor

Should not see any events except

t4_exp (hello tx timer expiry)

Non-zero (low number) error

counters are acceptable as long as

they do not increment (take 2-3

snapshots)



Troubleshooting VSL: LMPvss# sh switch virtual link portLMP summary

Link info: Configured: 2 Operational: 2

Peer Peer Peer Peer Timer(s)runningInterface Flag State Flag MAC Switch Interface (Time remaining)--------------------------------------------------------------------------------Te1/6/4 vfsp operational vfsp 0004.9bbe.ac00 2 Te2/6/4 T4(756ms)

T5(29.98s)Te1/6/5 vfsp operational vfsp 0004.9bbe.ac00 2 Te2/6/5 T4(756ms)

T5(29.92s)

Flags: v - Valid flag set f - Bi-directional flag sets - Negotiation flag set p - Peer detected flag set

Timers: T4 - Hello Tx Timer T5 - Hello Rx Timer

LMP Status

Last operational Current packet Last Diag Time sinceInterface Failure state State Result Last Diag

-------------------------------------------------------------------------------Te1/6/4 Link down Hello bidir Never ran --Te1/6/5 Link down Hello bidir Never ran --

LMP hello timer

Hello Tx (T4) ms Hello Rx (T5*) msInterface State Cfg Cur Rem Cfg Cur Rem-------------------------------------------------------------------------

Te1/6/4 operational - 1000 756 - 30000 29896Te1/6/5 operational - 1000 756 - 30000 29228

Compared to previous command

this one provides details of the

previous failure (if there was any) of

VSL links

Rest of the information is identical



Troubleshooting VSL:RRP

vss# sh switch virtual role detail

Switch Switch Status Preempt Priority Role Session IDNumber Oper(Conf) Oper(Conf) Local Remote

------------------------------------------------------------------LOCAL 1 UP FALSE(N ) 100(100) ACTIVE 0 0

REMOTE 2 UP FALSE(N ) 100(100) STANDBY 6480 9910

RRP Counters:--------------------------------------------------------------------

Inst. Peer Direction Req Acc Est Rsugg Racc----------------------------------------------------------------------1 1 Tx 0 1 0 1 31 1 Rx 2 0 1 0 3

RRP FSM info:--------------------------------------------------------------------sm(vslp_rrp RRP SM information for Instance 1, Peer 1), running yes, state role_resLast transition recorded: (lmac)-> lstart (req)-> hold (srt_exp)-> hold (req)-> hold(est)-> role_neg (srt_exp)-> role_neg (racc)-> role_res (racc)-> role_res (srt_exp)-> role_res (racc)-> role_res (srt_exp)-> role_res (srt_exp)-> role_res

In dual-active recovery mode: No

One of the switches must be

standby. If both are active it means

VSS has recovered from dual-

active condition, but new standby

has not been reloaded, most likely

due to unsaved config

This only refers to local switch



Troubleshooting VSL

vss# sh switch virtual link port-channelFlags: D - down P - bundled in port-channel

I - stand-alone s - suspendedH - Hot-standby (LACP only)R - Layer3 S - Layer2U - in use N - not in use, no aggregationw - waiting to be aggregated

Group Port-channel Protocol Ports------+-------------+-----------+-------------------10 Po10(RU) - Te1/6/4(P) Te1/6/5(P)20 Po20(RU) - Te2/6/4(P) Te2/6/5(P)

vss# ping vslp output interface t1/6/4 count 100 size 1388

Type escape sequence to abort.Sending 100, 1388-byte VSLP ping to peer-sup via output port 1/6/4, timeout is 2seconds:!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Success rate is 100 percent (100/100), round-trip min/avg/max = 12/12/28 ms

All ports on both sides of VSL

should be in bundled (P) state

Verify reliability of each individual

VSL link – output interface specifies

egress link (one of the VSL

interfaces). VSLP ping should work

when VSL is up, even if remote is in

RPR mode etc



Note: with VSS many commands use ‘switch module ’notation instead of just ‘module ’

In case of issues with VSL or VSS bring up, collect the followinginformation

sh tech(if VSS is split, collect from both sides)

remote command switch sh monitor event vslp all detail(if VSS is split, collect from both sides)

Troubleshooting VSL:what information to collect



VSS Agenda

Initialization


Spanning Tree

1st hop redundancy

Traffic forwarding




High AvailabilityRedundancy Mechanisms

The default redundancy mechanism between the 2 VSS chassis and their associated

supervisors is NSF/SSO, allowing state information and configuration to besynchronized. Additionally, only in NSF/SSO mode does the Standby supervisor PFC,

Switch Fabric, modules and their associated DFCs become active…

VSL

Should a mismatch of information occur between the Active and Standby Chassis, the

Standby Chassis will revert to RPR mode, where only configuration is synchronized, but

PFC, Switch Fabric and modules will not be brought up

Switch 1

Active

Switch 2

SSO Standby

VSL

Switch 1

12.2(33)SXI3

Active

Switch 2

12.2(33)SXH2

RPR Standby



In case of certain mismatches standby will only boot to RPR mode

(fabric, PFC & modules will be down)

vss# show switch virtual redundancyMy Switch Id = 1

Peer Switch Id = 2Last switchover reason = none

Configured Redundancy Mode = ssoOperating Redundancy Mode = rpr

...vss# show switch virtual redundancy mismatch

Startup Config Mismatch:Mismatch in config file between local Switch 1 and peer Switch 2:ACTIVE : Interface TenGigabitEthernet1/6/5 shutdownSTANDBY : Interface TenGigabitEthernet1/6/5 not shut

Other possibilities

IOS version mismatch

Other VSL-related config mismatch

Non-SSO redundancy mode is configured

Forwarding engine (PFC) mismatch

Troubleshooting redundancy:why standby is not in SSO mode



VSS with 4 supervisors

Initially in-chassis redundantsupervisors were kept in rommon not used

As of 12.2(33)SXI4 in-chassisredundant supervisors function asa linecard – ports are useable

Before switching to linecard modesupervisors will boot to RPR-warm

mode meaning they will have theirconfiguration synchronized

If active supervisor fails entirechassis is reloaded 2nd chassistakes over same model as with2 sups

If supervisor fails completely(doesn’t boot) or removed, the in-chassis redundant supevisor willboot as active supervisor noneed to follow procedure forsupervisor replacement

VSL

SiSi SiSi

Active SSO

rommon> rommon>

VSL

SiSi SiSi

Active SSO

RPR-warm RPR-warm

Pre-12.2(33)SXI4

12.2(33)SXI4 and later



What is Dual-Active?

If VSL goes down standby needsto know if it was just VSL or the

active switch that failed

For faster failovers assumption isthat active switch fails Oldstandby becomes Active a.s.a.p.

If old Active is still there however we will have 2 devices withidentical config on the network

IGP adjacencies will start to flapor will go down

L2 MEC will be error-disabledafter ~1 minute by EtherChannelmisconfig guard (because ofreceiving 2 different BPDUs)

VSLSiSi SiSi

Active Standby

SiSi

Active

Dual-active, if not detected will cause severe network outage

Configure robust dual-active detection

Layer2-MEC

Layer3-MEC


21/124


Dual-Active Detection options

Enhanced PAGP

Hot Standby Active

Switch 1 Switch 2

IP-BFD

Switch 1

VSLP VSLP BFD BFD

Switch 2

Hot Standby Active

Switch 1 Switch 2

Hot Standby Active

VSLP Fast Hello

L2 Heart Beat Link

Software-12.2(33)SXI

Enhanced subsecond detection in

12.2(33)SXI3

L3 Heart Beat Link

Software -12.2(33)SXH1

Requires PAGP+ capable neighbor with

• 375012.2(46)SE

• 450012.2(44)SE

• 650012.2(33)SXH

Software -12.2(33)SXH1


22/124


Dual Active Recovery

Switch 1 detects that switch 2 is now also active triggering dual active

condition thus switch 1 brings down all the local interfaces to avoid networkinstability. Until VSL link restoration occurs, switch 1 is isolated from thenetwork;

Once the VSL link comes up, the role negotiation determines that switch 1needs to come up in STAND_BY mode hence it reboots itself; finally, allinterface on switch 1 are brought on line and switch 1 assumes STAND_BY

role

Switch 1All

InterfacesDown

Dual Active Recovery

Switch 1Reboot and

Comes Up inSTAND_BY

Mode

VSS Restoration

Switch 2 inACTIVEMode

OLDACTIVE

NewACTIVE


23/124


If configuration was changed but has not been saved the would-be-standby switch will not be reloaded following VSL recovery

Save the config & reload standby

19:54:59: %VSLP-SW2_SP-5-RRP_MSG: Role change from Active to Standby and hence need

to reload19:54:59: %VSLP-SW2_SP-5-RRP_UNSAVED_CONFIG: Ignoring system reload since there areunsaved configurations. Please save the relevant configurations

19:54:59: %VSLP-SW2_SP-5-RRP_MSG: Use 'redundancy reload shelf' to bring this switchto its preferred STANDBY role

Dual-active recovery, …

Reload from active switch will not correct this

After reloading it might happen that config between Active and Standbyis not consistent Standby will come up in RPR modeSave the config once again and reload standby again (redundancyreload peer)


24/124


Virtual Switching SystemWhich Dual Active Recovery Method Should I Use?

Since dual-active detection is importantredundancy is highly recommended

Use Fast-hello + e-PAgP

In case of all-LACP deployment, use Fast-hello over port-channel

Only case where BFD had advantage was inpre-SXI3 release with routed ECMP uplinks

and OSPF

SiSiSiSi

RedundantVSL Fiber

ePAgP

ePAgP

VSLP Fast-Hello

or BFD


25/124


VSS Agenda

Initialization


Spanning Tree

1st hop redundancy

Traffic forwarding



26/124


Spanning Tree and VSS

STP process

Active Standby

VSS domain behaves as a single bridge

STP runs only on SP of active switch

VSL is not part on STP and will not be blocked

BPDUs will travel across single link of the MEC

STP will be blocking ports is there are redundant

links Keep STP enabled

Physical Logical

1

2

3

4

1 2

3

4


27/124


Troubleshooting STPvss#sh spanning-tree interface po201 detail

Port 5767 (Port-channel201) of VLAN0001 is designated forwarding

Port path cost 3, Port priority 128, Port Identifier 128.5767.

Designated root has priority 0, address 001e.4963.7b94

Designated bridge has priority 32768, address 0008.e3ff.fdbdDesignated port id is 128.5767, designated path cost 16

Timers: message age 0, forward delay 0, hold 0

Number of transitions to forwarding state: 1

Link type is point-to-point by default

BPDU: sent 4447, received 12

...

vss# remote login switch

vss-sp# debug interface po201Condition 1 set

vss-sp# debug spanning-tree switch tx

Spanning Tree Switch Shim transmit bpdu debugging is on

Dec 6 14:59:22.594: SW1_SP: STP SW: FAST TX: VLAN 555 Port-channel201: bpdu size 116, refcnt 1




vss-sp# debug spanning-tree switch tx decodeSpanning Tree Switch Shim decode transmitted packets debugging is on

Dec 6 14:59:43.510: SW1_SP: STP SW: FAST TX: 0180.c200.0000


28/124


Spanning Tree stability features recap

Feature Condition Works on Effect Note

UDLD

Detects if link becomes

unidirectionalI.e. link cannot carry BPDUs

both ways causes loops

Physicalport

Error-disables

unidirectionallinks

Useful on port-channels to

take out broken links,alternative fast-timers

PAGP/LACP

Bridge

Assurance

(BA)

Expects to receive a BPDU

every hello_time from the

peer.

I.e. cases of dead control

plane on the remote side,also BPDU loss

Logical

port

Blocks port at

STP level

(BA-

inconsistent

state)

Main protection mechanism

where supported, alternative

is Loop Guard

Dispute

Checks the remote port role

in the received BPDU, role

should not be designated in

BPDU received on

designated port

Cases of unidirectionalcommunication

Logical

port

Blocks port at

STP level

(Disputed

state)

Complements BA, on by

default. Somewhat overlaps

with UDLD, but not as

effective on port-channels.

Only works with RSTP/MST

BPDUs

Loop

Guard

Doesn’t allow port to takedesignated role if it stopped

receiving BPDUs

Unidirectional

communication, control plane

issues on remote

Logical

port

Blocks port at

STP level

(Loop-

inconsistent)

Superseded by BA + Dispute,

use with PVST+ or when BA

is not supported


29/124


Bridge assurance, Dispute & UDLD

Preferred combination is Bridge Assurance + UDLD normalmode + Dispute (on all interswitch links) when both sidessupport it

UDLD is needed to take out bad links from port-channels(otherwise BA or Dispute will keep whole port-channelblocked). PAgP/LACP will take out bad links, but will takelonger (~105sec vs ~20sec for UDLD with 7 sec timer)

If preferred config is not supported use Loop Guard + UDLD

(supported by all Cisco switches)

Defaults: BA/UDLD – disabled, Dispute - enabled


30/124


VSS Agenda

Initialization


Spanning Tree

1st hop redundancy

Traffic forwarding



31/124


Asymmetric Routing

Alternating HSRP Active betweendistribution switches can be usedfor upstream load balancing,however downstream traffic hitsboth distribution block switches

This can cause a problem

with unicast flooding ARP entries age in 4 hours while

L2 entries age in 5 minutes

ARP entry with no matching L2entry unicast flooding

In many cases when the HSRPstandby needs to forward a frameit will have to unicast flood theframe since it’s CAM table isempty VLAN 2

SiSiSiSi

VLAN 3

Switch 1: Active

HSRP and Root

Bridge VLAN 3

VLAN 2VLAN 3

Switch 2: Active

HSRP and Root

Bridge VLAN 2

CAM Table

Empty for

VLAN 2

CAM Table

Empty for

VLAN 3

B

BB

B

B

With VSS there is single logical router thus no asymmetric routing


32/124


1st hop redundancy with VSS

MAC_A Router MAC

IP A IP B

Router MAC

0001.0002.0003

Router MAC

0001.0002.0003

MAC_B Router MAC

IP B IP A

PC A

PC BVSS acts as 1 router there is 1 router MAC

address, both switches will L3 switch packets

destined to that MAC address

Once either switch learns dynamic MAC address,

other switch will also learn no unicast floods

due to asymmetry of traffic between switches

In case of failover router MAC address does not

change Inherrent 1st hop redundancy


33/124


VSS mac-address

By default VSS will use Router mac-address from active switch backplane

Router mac-address is maintained across switchovers – no 1st hop redundancyprotocol is needed

If entire VSS system is brought down and then up again and switch 2 ends upbeing active – router mac-address might change (this will only have impact ondevices that ignore gratuitous ARPs)

To avoid such change, use ‘mac-address use-virtual’ – with this command VSS willuse special mac-address reserved for VSS

vss(config)#switch virtual domain 111vss(config-vs-domain)#mac-address use-virtual

Configured Router mac address is different from operational value. Change will takeeffect after config is saved and the entire Virtual Switching System (Active andStandby) is reloaded.

Virtual mac is based on 0008.e3ff.fc00

Alternatively router-mac maybe statically configured with ‘mac-address’ in the domain config context


34/124


Troubleshooting Router-MAC

vss# sh interface vlan 226

Vlan226 is up, line protocol is up

Hardware is EtherSVI, address is 0008.e3ff.fdbc (bia 0008.e3ff.fdbc)

Internet address is 192.168.222.18/30

...

vss# sh mac-address-table address 0008.e3ff.fdbc vlan 226 allLegend: * - primary entry

age - seconds since last seen

n/a - not available

vlan mac address type learn age ports

------+----------------+--------+-----+----------+--------------------------

Supervisor switch 1 Module 6

* 226 0008.e3ff.fdbc static No - Router

Supervisor switch 2 Module 6* 226 0008.e3ff.fdbc static No - Router

vss# sh mac-address-table address 0008.e3ff.fdbc vlan 226 detail switch 2 module 6

MAC Table shown in details

========================================PI_E RM RMA Type Alw-Lrn Trap Modified Notify Capture Flood Mac Address Age Pvlan SWbits Index XTag

----+---+---+----+-------+----+--------+------+-------+------+--------------+----+------+------+------+----


Yes No No ST No No No No No No 0008.e3ff.fdbc 0xE8 226 0 0x380 1

What is router MAC for given

interface

It should be pointing to the ‘Router’

Actual hardware L2 entry must

have non-zero Xtag in order forforwarding engine to consider such

packets for L3 switching

When VSS receives a packet destined to Router-MAC it will try to L3 switch

(route in hardware) the packet, else the packet will be bridged


35/124


MAC address learning with VSS

A ↓ A ↓

PC A

PC B

MAC A is learned on lower MEC, triggering theframe to be sent to every forwarding engine

(DFC/PFC) Flood to Fabric mechanism (HW)1

Internal frame header (carried over VSL) includes

source index which identifies source port and

hence the MAC is learned on lower MEC although

the frame is received on VSL

Depending on how traffic is flowing through VSS

some forwarding engines might not see the

packets from A after initial flood to fabric which

might lead to aging of address and flooding

MAC synchronization feature keeps address fromexpiring as long as traffic from that address is

seen anywhere in the system

1

2

2


36/124


MAC address synchronization Initial new learns are syncronized between switch 1 and switch 2

However if only switch 1 or switch 2 ‘sees’ the traffic for given address L2 entry might age out

in one of the switches (this behavior is per forwarding engine: PFC/DFC) In order to reduce chance of unicast flooding we need to keep L2 entries consistent access

both switches

‘mac-address-table synchronize’ feature will keep L2 tables synchronized

Enabled by default when WS-X6708 linecard is present in the chassis

Enabled by default in VSS as of 12.2(33)SXI4

Recommended in all cases

Make sure there is at least 2x aging intervals in synchonization interval(i.e. for sync interval 160, L2 aging is >320 seconds, 480 recommended)

vss(config)# mac-address-table synchronize

% Current OOB activity time is [160] seconds

% Recommended aging time for all vlans is atleast three times the activity intervaland global aging time will be changed automatically if required

When troubleshooting unicast flooding, 2 items are very important

What module traffic arrives to (use commands to check ether-channel load-balancing)

Whether the module in question has the mac-address learned

(use ‘sh mac-address address all’)


37/124


VSS Agenda

Initialization


Spanning Tree

1st hop redundancy

Traffic forwarding



38/124


Ingress forwarding model

Distributed architecture. Ingress forwarding engine makesforwarding, ingress *and* egress ACL/QOS decisions

IMPORTANT: If the linecard where packet is received has DFC –entries on that linecard need to be looked at when troubleshooting.

Otherwise look at active supervisor’s forwarding entries i.e. ‘sh mls cef module ’

or ‘sh mls cef ’

DFC DFC

Ingress EgressXFabric

Traffic flow


39/124


Traffic locality

Main concept for traffic forwarding is locality – Only local ports are used to send traffic out

– … except when there are no local ports, this is when traffic will crossVSL/Peer-link


40/124


Traffic locality for ECMP routes

ECMP follows a similar behavior, locallinks are preferred and all traffic isforwarded out of a locally attached link

Hardware FIB inserts entries for ECMProutes using locally attached links

If all local links fail the FIB is programmedto forward across the VSL link

vss# sh ip route 10.121.0.0 255.255.128.0 longer-prefixes

D 10.121.0.0/17[90/3328] via 10.122.0.33, 2d10h, TenGigabitEthernet2/2/1

[90/3328] via 10.122.0.27, 2d10h, TenGigabitEthernet1/2/1[90/3328] via 10.122.0.22, 2d10h, TenGigabitEthernet2/2/2[90/3328] via 10.122.0.20, 2d10h, TenGigabitEthernet1/2/2

vss# sh mls cef 10.121.0.0 17 switch 1

Codes: decap - Decapsulation, + - Push LabelIndex Prefix Adjacency102400 10.121.0.0/17 Te1/2/2 , 0012.da67.7e40 (Hash: 0001)

Te1/2/1 , 0018.b966.e988 (Hash: 0002)

Four ECMPEntries

Two FIB

Entries

Te1/2/2

Te1/2/1

SW1

SiSi SiSi


41/124


Important:: Only use parameters

consistent with the configured

load-balancing algorithm.

Command uses all the specified

arguments to calculate the hash.

VSS L2/L3 Forwarding (Data Plane)

Identify the physical path for flow from host 2 host 1 (out of Port-channel 2)

vss# show etherchannel load-balance hash-result interface Port-channel 2 switch 1ip 9.0.1.2 vlan 705 8.0.1.1

Computed RBH: 0x6Would select Gi1/6/2 of Po2

vss# show etherchannel load-balance hash-result interface Port-channel 2 switch 2ip 9.0.1.2 vlan 705 8.0.1.1

Computed RBH: 0x6

Would select Gi2/9/15 of Po2

Packet coming in on switch 1, needing to goout on Po2 will select Gi1/6/2

Packet coming in on switch id 2, needing to

go out on Po2 will select Gi2/9/15

Verify the load-balance algorithm usedvss# show etherchannel load-balance switch 2 module 2

EtherChannel Load-Balancing Configuration:

src-dst-ip vlan included

mpls label-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source XOR Destination MAC address

IPv4: Source XOR Destination IP addressIPv6: Source XOR Destination IP address

MPLS: Label or IP

VSS Data Plane Troubleshooting L2 MECVSS specific commands

augmented with switch id


42/124


VSS L2/L3 Forwarding (Data Plane)

Routing table shows two Equal Cost Paths to 9.0.0.0/8vss# show ip route 9.0.0.0 | i via

Known via "eigrp 101", distance 90, metric 3072, type internal

Redistributing via eigrp 101

7.7.1.2, from 7.7.1.2, 1d00h ago, via TenGigabitEthernet2/2/7

* 7.6.1.2, from 7.6.1.2, 1d00h ago, via TenGigabitEthernet1/3/2

Looking at the HW table shows next hop directly attached to local switchis preferred

vss# show mls cef lookup 9.0.1.0 switch 1 mod 3

Codes: decap - Decapsulation, + - Push Label

Index Prefix Adjacency

108775 9.0.0.0/8 Te1/3/2 , 000f.35ed.7c00

vss# show mls cef lookup 9.0.1.0 switch 2 mod 2

Codes: decap - Decapsulation, + - Push Label

Index Prefix Adjacency

108775 9.0.0.0/8 Te2/2/7 , 000f.35ed.7c00

DUT# show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 1 mod 3

Interface: Te1/3/2, Next Hop: 7.6.1.2, Vlan: 4064, Destination Mac: 000f.35ed.7c00

DUT# show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 2 mod 2

Interface: Te2/2/7, Next Hop: 7.7.1.2, Vlan: 4056, Destination Mac: 000f.35ed.7c00

Packet coming in on switch 1 module 3, for 9.0.0.0/8

prefers next hop attached to local switch id 1

Packet coming in on switch 2 module 2, for 9.0.0.0/8

prefers next hop attached to local switch id 2

VSS Data Plane Troubleshooting ECMP: Host 1 Host 2


43/124


vss# sh mac-address-table address 0005.9a3b.6c80 vlan 226

...

vlan mac address type learn age ports

------+----------------+--------+-----+----------+--------------------------


* 226 0005.9a3b.6c80 dynamic Yes 10 Po3

Supervisor switch 2 Module 6* 226 0005.9a3b.6c80 dynamic Yes 10 Po3

vss# sh etherchannel 3 summary

...

Group Port-channel Protocol Ports

------+-------------+-----------+-----------------------------------------------

3 Po3(SU) PAgP Gi1/1/15(D) Gi2/6/3(P)

VSS

Po4

What is the port for this mac

address

What are physical ports of port-

channel

All ports on switch1 side aredown

If packet will arrive to switch1 to

be switched to po3, packet will

cross VSL

Po3

1/1/33

2/4/33

1/1/15

2/6/3

0005.9a3b.6c80

Will thepacket crossVSL link?


44/124


vss# sh mac-address-table address 0005.9a3b.6c80 vlan 226 detail switch 1 module 6MAC Table shown in details

========================================PI_E RM RMA Type Alw-Lrn Trap Modified Notify Flood Mac Address Age Pvlan Index XTag----+---+---+----+-------+----+--------+------+------+--------------+----+------+------+----Supervisor switch 1 Module 6Yes No No DY No No Yes No No 0005.9a3b.6c80 0x86 226 0xB40 0

vss# remote command switch test switch virtual ltl index 0xB40...

Unmapped index: 0xB40------+----------------------------------------SW viewIndex | Ports------+----------------------------------------0x0B40 Po3[Gi2/6/3],Po10[Te1/6/4]...------+----------------------------------------HW view

Index | Ports------+----------------------------------------0x0B40 Te1/6/4,Gi2/6/3...

vss# sh switch virtual link port-channel | i PoGroup Port-channel Protocol Ports10 Po10(RU) - Te1/6/4(P)20 Po20(RU) - Te2/6/4(P)

VSS

Po4

Find the index for given mac

address on ingress forwarding

engine

Find what ports on the local

switch (1) this index includes Index should include VSL ports

How to verify if the packet from

switch 1 will cross VSL in order to

reach next-hop mac-address?

Po3

1/1/33

2/4/33

1/1/15

2/6/3

0005.9a3b.6c80

Will thepacket crossVSL link?

VSS f di t bl h ti


45/124


VSS forwarding troubleshootingsummary

Unless the traffic is crossing VSL,troubleshooting VSS packet forwarding isexactly the same as troubleshootingstandalone cat6500

When traffic crosses VSL, verify

– L3 entries on the ingress forwardingengine (PFC or DFC)

– L2 entries (for next hop destination mac)on forwarding engine servicing the VSL onthe 2nd chassis (strictly speaking L2 entriesneed to be checked on all DFCs along thepacket path)


46/124


Special case for flooding

MAC_A

MAC_B

MAC B is not known flood the frame11

Internal frame header (carried over VSL) includes

destination index which is remapped by egress

switch to another index that does not include any

MEC that has operational ports on ingress switch

2

Frame is flooded to devices that are single

connected to egress switch (on the right)

3

2

3


47/124


Each flow is assigned to 1 of 8 ‘buckets’ Each port in port-channel transmits traffic for some buckets (i.e. 4 for 2-port channel, 2

for 4-port etc) When ports are joining/leaving channel the buckets are redistributed among operational

ports in deterministic fashion

Flows that remain on operational ports might be disturbed while ASICs are beingprogrammed

With adaptive hash option, only buckets that must move are reprogrammed

Member 1 Member 2

1 23 4

5 6

7 8

Member 1 Member 2 Member 3

1 2 34 5 6

7 8

New member

joins

EtherChannel Adaptive Hash

Member 1 Member 2

1 2

3 4

5 6

7 8

Member 1 Member 2 Member 3

1 2 3

5 4 6

7 8

New member

joins

buckets that must move

buckets moving betweenoperational ports

buckets that must move


48/124


Adaptive hash is enabled by default on VSL link

If there is 1 link / chassis / MEC – adaptive hash on MEC will not make any difference

If the network consists of several adjacent VSS systems, adaptive hash was enhancedto avoid traffic polarization (as of 12.2(33)SXI)

Configured per port-channel

With adaptive hash less flows should be impacted when ports join or leave port-channels

This is mostly evident when control-plane is busy (i.e. when many changes arehappening at the same time – during failovers etc)

EtherChannel Adaptive Hash

vss(config)#int port-channel200

vss(config-if)#port-channel port hash-distribution adaptive


49/124


SPAN

When SPAN’ed traffic is crossing VSL it is transmittedover single link this might cause oversubscription ofVSL link if amount of SPAN’ed traffic is significant

Use MEC as SPAN destination to prevent SPAN’edtraffic from crossing VSL

If one side of the MEC goes down – SPAN’ed traffic willcross VSL

Provision enough bandwidth on VSL

Use ‘port-channel min-links’ LACP feature on SPAN

destination MEC to bring down MEC if link is down on oneside

Use EEM script to shut down MEC or SPAN session whenone side of SPAN destination MEC goes down


50/124


VSS Agenda

Initialization


Spanning Tree

1st hop redundancy

Traffic forwarding



51/124


Multicast forwarding

Layer 2 access has two multicast routers on the access subnet, RPFchecks and split roles between high and low IP address routers

VSS has a single multicast router which simplifies multicast topology

The multicast forwarder is selected based on which member of VSSlink receives multicast traffic

SiSi

Designated

Router (High IP Address)

IGMP Querier (Low IP address)

Non-DR Has to

Drop AllNon-RPF Traffic

SiSi

Single Logical MulticastDesignated Router and IGMP Querier

MEC behavior upon


52/124


MEC behavior uponVSS recovery after SSO switchover

vss(config)#port-channel load-defer 120vss(config)#int po200vss(config-if)#port-channel port load-deferThis will enable the load share deferral feature on this port-channel.The port-channel should connect to a Virtual Switch (VSS).Do you wish to proceed? [yes/no]: y

To prevent this issue, configure ‘port-channel load-defer ’ feature on upstream switch

Upstream switch will delay sending traffic to newly bundled port for configured duration

Following SSO switchover left switch comes up

after reload

1

MEC link from left switch is brought up and joins

the bundle

2

Top switch starts sending a share of traffic to the left

switch, but the left switch might still be converging

(loading FIB tables, programming ASICs etc), so itmight not be fully ready to correctly forward the this

traffic

this might cause part of traffic to be lost for

some time after the switch recovery

3

1

2

3


53/124


Multicast fast-redirect

When a member of egress

Layer2 port-channel (MEC orDEC) is unbundled/bundled OnVSS replicating multicast traffic inegress mode it might takenoticeable time to reprogramhardware to send traffic via

remaining links (local or acrossVSL)

Fast-redirect feature shortensreprogramming time bypreprogramming most of theneeded changes

SiSi SiSi

MEC

MEC

Sources

Receivers

vss(config)#interface port-channel 40vss(config-if)#mls ip multicast egress fast-redirect

VSS s mmar


54/124


VSS: summary

1 active redundant control plane

single config single point of management


Standby switch is essentially a

set of additional linecards

Control messages and Data

frames flow between active and

standby via VSL(can be seen as backplane

extension)

Special encapsulation on VSL

frames to carry additional

information

Active

Data Plane

ActiveControl Plane

Active

Data Plane

StandbyControl Plane

MEC

VSL

Dual-Active

detection link

Active Standby

VSS domain


55/124

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKCRS-1930

VPC


56/124


Both VPC and VSS

• simplify logical Layer 2 topology• use Traffic Locality for efficient shortest pathforwarding


57/124


VPC Agenda

Initialization

Redundancy considerations

Spanning Tree

Traffic forwarding

1st hop redundancy


VPC Virtual Port channel


58/124


VPC – Virtual Port channel 2 active control planes

2 configs

2 points of management


Primary-Secondary notion for some

aspects of operation

Control messages and Data framesflow between active and standby via

Peer-Link

Peer-Link is L2 trunk with plain 802.1q

encapsulation

Control messages are carried by CFS

over Peer Link

Peer keepalive link to detect dual-

active condition

We call VPC the MCEC between VPC

domain and access switches

ActiveData Plane

ActiveControl Plane

ActiveData Plane

ActiveControl Plane

VPC

Peer-Link

Peer Keepalive link

Primary Secondary

VPC domain


59/124


VPC initialization

VPC init is largely independent of NXOS boot eachswitch boots on its own

VPC feature starts

Keep-alive linkup / peer communication is established

Peer-link linkup / CFS communication is established

Primary/Secondary role is resolved

Consistency is checked via CFS and applications synced

Peer-Link brought UP for data

VPCs brought UP

Cisco Fabric Services


60/124


Nexus# sh cfs application----------------------------------------------Application Enabled Scope----------------------------------------------arp Yes Physical-ethstp Yes Physical-ethvpc Yes Physical-ethigmp Yes Physical-ethl2fm Yes Physical-eth

...

Cisco Fabric ServicesCFS

Uses

• Configuration validation

• MAC member port synchronization

• vPC member port status

• IGMP snooping synchronization

• vPC status

For VPC CFS messages are encapsulated in Ethernet framesdelivered between peers on the peer-link

CFS messaging

VPC Configuration consistency


61/124


VPC has distributed management plane. Configurations of bothswitches are managed separately

Some configurations inconsistencies could lead to undesirableforwarding implications (packet duplication, blackholing etc). VPCtakes different action depending on the type of inconsistency

Type 1: VPC will not come up

Type 2: VPC will come up, but undesirable forwarding implications

might occur, syslog will be printed upon detected inconsistency

VPC Configuration consistency

Nexus# sh vpc consistency-parameters interface port-channel 1Name Type Local Value Peer Value------------- ---- ---------------------- -----------------------lag-id 1 [(7f9b, [(7f9b,...mode 1 active activeSTP Port Type 1 Default Default

STP Port Guard 1 None NoneSTP MST Simulate PVST 1 Default DefaultNative Vlan 1 1 1Port Mode 1 trunk trunkMTU 1 1500 1500Duplex 1 full fullSpeed 1 10 Gb/s 10 Gb/sAllowed VLANs - 101 101

Nexus# sh vpc consistency-parameters globalName Type Local Value Peer Value------------- ---- ---------------------- -----------------------STP Mode 1 Rapid-PVST Rapid-PVSTSTP Disabled 1 None NoneSTP MST Region Name 1 "" ""STP MST Region Revision 1 0 0

STP MST Region Instance to 1VLAN MappingSTP Loopguard 1 Disabled DisabledSTP Bridge Assurance 1 Enabled EnabledSTP Port Type, Edge 1 Normal, Disabled, Normal, Disabled,BPDUFilter, Edge BPDUGuard Disabled DisabledSTP MST Simulate PVST 1 Enabled EnabledInterface-vlan admin up 2 101 101

Interface-vlan routing 2 1,101 1,101

T bl h ti VPC i iti li ti


62/124


Troubleshooting VPC initialization Use sh vpc to check the feature status

vpc1# show feature | i vpc

vpc 1 enabledvpc1# sh vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1

Peer status : peer adjacency formed ok

vPC keep-alive status : peer is alive

Configuration consistency status: success

Type-2 consistency reason : Consistency Check Not Performed

vPC role : primary

Number of vPCs configured : 1

Peer Gateway : Disabled

Dual-active excluded VLANs : -

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 Po100 up 1,101

vPC status

----------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

-- ---- ------ ----------- ------ ------------

1 Po1 up success success 101

CFS can communicate with the

peer We hear peer-alives

Configs are compatible

Master/Slave for certain apps

Peer-Link will come up after CFS +

Peer-Keepalive + Config check are

ok

T bl h ti VPC i iti li ti


63/124


Troubleshooting VPC initialization

Stable, not expecting issues here

Set VPC logging level to 5 (default) to see more verbose messaging during theVPC bringup

vpc1(config)# logging level vpc 5

08:18:47 %ETHPORT-5-SPEED: Interface port-channel100, operational speed changed to 10 Gbps Peer-Link comes up

08:18:51 %VPC-3-PEER_UNREACHABLE: Remote Switch Unreachable

08:18:51 %VPC-3-VPC_PEER_LINK_BRINGUP_FAILED: vPC peer-link bringup failed (vPC peer is not reachable over cfs)

08:18:51 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 1,100-101 on Interface port-channel100 are being suspended.(Reason: vPC peer is not reachable over cfs)

08:18:51 %ETHPORT-5-IF_UP: Interface port-channel100 is up in mode trunk

08:18:58 %VPC-4-VPC_ROLE_CHANGE: In domain 1, VPC role status has changed to primary

08:18:58 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 1,100-101 on Interface port-channel100 are removed fromsuspended state.

08:18:58 %VPC-5-VPC_DELAY_SVI_BUP_TIMER_START: vPC restore, delay interface-vlan bringup timer started

08:19:08 %VPC-5-VPC_DELAY_SVI_BUP_TIMER_EXPIRED: vPC restore, delay interface-vlan bringup timer expired,reiniting interface-vlans

08:19:08 %VPC-5-VPC_RESTORE_TIMER_START: vPC restore timer started to reinit vPCs

08:19:38 %VPC-5-VPC_RESTORE_TIMER_EXPIRED: vPC restore timer expired, reiniting vPCs

In case process does not go beyond certain stage, one should look atcommunication between the peers (CFS)


64/124


VPC config remarks

Check config consistency using ‘sh vpc consistency-parameters’

Complete list of parameters which should be consistent is quiteextensive: physical port config, QOS, security, STP, routingprotocols etc

check config guide for specific NXOS version

Domain id must be unique for each domain reachable adjacentat Layer 2

VPC domain 100

VPC domain 200

VPC

Domain id MUST be

different

(can’t be 100 on bothPair)


65/124


VPC: CFS troubleshooting

Cisco Fabric Services Transport of control messages between VPC peers

Nexus# show cfs status

Distribution : Enabled

Distribution over IP : Disabled

IPv4 multicast address : 239.255.70.83

IPv6 multicast address : ff15::efff:4653

Distribution over Ethernet : Enabled

Nexus# show cfs peers

Physical Fabric

---------------------------------------------

Switch WWN IP Address

---------------------------------------------

20:00:00:1b:54:c2:42:41 10.48.73.222 [Local]

Nexus

20:00:00:1b:54:c2:42:44 0.0.0.0

Total number of entries = 2

Nexus# show cfs internal ethernet-peer statistics| i Trans|Rece

Number of Segments Transmitted : 218

Number of Acks Transmitted : 223

Maximum Segment Size Transmitted : 0

Number of Transmission Timeouts : 0

Number of segments in Transmit Queue : 0

Number of segments in Re-Transmit Queue : 0

Total Number of Segments Received : 441

Number of Acks Received : 217

Number of Duplicate Messages Received : 0

Number of Unexpected Segments Received : 0

Number of fragmented segments Received : 2

Number of duplicate fragments Received : 0

Number of unfragmented segments Received : 210

Number of Received Segments Dropped : 0

Number of Unreliable segments Transmitted : 1Number of Unreliable segments Received : 1

Nexus# sh cfs internal notification log name vpc

Sun Nov 14 15:27:22 2010: Peer add 20:00:00:1b:54:c2:42:44

Sun Nov 14 19:05:25 2010: Peer gone 20:00:00:1b:54:c2:42:44

Sun Nov 14 19:08:03 2010: Peer add 20:00:00:1b:54:c2:42:44

TX/RX counters should move whenVPC is active or coming up

Remote peer should be seen

Shows timestamps for when CFS

communication for VPC was

interrupted (peer-reload, peer-link

issues etc)


66/124


More information

sh tech(collect for offline analysis, takes ~5 min when redirected to file)

sh tech vpc(collect when there is no time for ‘big’ sh tech)

debug vpc peer (peer events, useful for indepth vpc troubleshooting)

debug vpc peer-link(peer-link events, for indepth vpc bringup troubleshooting)

debug cfs event ethernet

(cfs event – peer communication)


67/124


VPC Agenda

Initialization


Spanning Tree

Traffic forwarding

1st hop redundancy


VPC redundancy model


68/124


Process restartability

Supervisor redundancy

VPC redundancy

Active

Standby(SSO)

Active

Standby(SSO)

Process 1

Process 2

Process X

…

Process 1

Process 2

Process X

…

Switch 1 Switch 2

VPC Domain

Processes checkpoint their runtime state Crashing process is restarted statefully by

system manager

HA-policy will triggersupervisor switchover

in response to

excessive process

crashing, software,

hardware or

diagnostic failure

VPC redundancy model

Devices dual-attached to VPC domain are protected against

single switch failure (power, hardware, maintenance etc)

Peer-link failure handling


69/124


Peer link failure handling(similar to dual-active detection in VSS)

VPC peer-link failure

I am primary

Done

ReceivingKeepalives*

Bring down all VPC ports

Become primary

primary

2ndary

yes

no

Primary is alive

Primary is gone

VPC peers do not require reload following

peer-link failure or recovery


70/124


Keepalive link

Heartbeat between vPC peers to prevent dual-active scenario

Keepalives are sent every second by default on UDP port 3200

3 second hold timeout on peer-link loss (ignore keepalive to leavetime for convergence before taking action)

5 seconds keepalive timeout (starts after hold timeout after peer-linkdown) – if no keepalive received during this timeout dual active

detection seconday bring down VPC Use dedicated link, though NXOS does not enforce this – just IP

connectivity is verified

Mgmt interface can be used as keepalive link, but do not connect themanagemet interfaces together directly (only active supervisormanagement interface is up)

vpc1# debug vpc peer-keepalive13:10:54.257099 vpc: received new OOB packet, version(0) flags(0) my_context(0)your_context(0) my_epoch(604049) your_epoch(604104) my_ip(1.1.1.2)13:10:54.257126 vpc: your_ip(1.1.1.1) domainId(1)13:10:55.257442 vpc: received new OOB packet, version(0) flags(0) my_context(0)your_context(0) my_epoch(604050) your_epoch(604105) my_ip(1.1.1.2)13:10:55.257469 vpc: your_ip(1.1.1.1) domainId(1)13:10:56.257324 vpc: received new OOB packet, version(0) flags(0) my_context(0)your_context(0) my_epoch(604051) your_epoch(604106) my_ip(1.1.1.2)13:10:56.257351 vpc: your_ip(1.1.1.1) domainId(1)

Peer Keepalives

Troubleshooting VPC peer-keepalives


71/124


Troubleshooting VPC peer keepalives

Nexus# show vpc peer-keepalive


--Send status : Success

--Last send at : 2009.06.19 00:41:15 589 ms

--Sent on interface : Eth2/35

--Receive status : Success

--Last receive at : 2009.06.19 00:41:14 580 ms

--Received on interface : Eth2/35

--Last update from peer : (1) seconds, (9) msec

vPC Keep-alive parameters--Destination : 7.7.7.77

--Keepalive interval : 1000 msec

--Keepalive timeout : 5 seconds

--Keepalive hold timeout : 3 seconds

--Keepalive vrf : v1

--Keepalive udp port : 3200

--Keepalive tos : 192

Nexus# show vpc statistics peer-keepalive


vPC keep-alive statistics

----------------------------------------------------

peer-keepalive tx count: 9773

peer-keepalive rx count: 8985

average interval for peer rx: 991

Count of peer state changes: 0

Peer-keepalive is only essential at

the time when peer-link goes down At any other time peer-keepalive

failure will only trigger syslog

Peer-keepalives might be affected

by extreme control plane load

(check CPU utilization & COPP)

Number of keepalive state

transitions, closer to 0 - better

VPC behavior at initialization


72/124


C be a o at t a at o(default)

VPC needs to be able to talk to the

peer (over peer-link) before bringingup VPC port-channels Negotiate LACP/STP operating roles for

the chassis

Wait for per-port peer parameters andhandshake to bring up vPC ports

Performs peer parameters consistencycheck on each VPC bringup

Only after VPC port-channels arebrought up.

What if after a full DC outage (bothNexus down), only one switch is comingup ?

Will not bring up VPCs if after adatacenter outage, only one VPC peercomes back up


73/124


VPC Reload Restore

Allows to bring up VPCs after timeoutif peer is presumed dead

Default timeout 240 sec

Assumes primary role for STP andLACP

Nexus(config)# vpc domain 1Nexus(config-vpc-domain)# reload restore ?

delay Duration to wait before assuming

peer dead and restoring vpcs

Nexus(config-vpc-domain)# reload restore delay ? Time-out for restoring vPC links

(in seconds)


74/124


ARP synchronization

PC A

PC B

ARP

Ip B Mac B

ARPIp B ???

Needs to be

Resolved ?

When traffic patternchanges (due to VPC linksgoing up/down, due tofailover etc) the peer thathandles the traffic mightneed to resolve ARP beforebeing able to forwardpackets

This might introduceadditional delay to trafficrecovery

ARP sync feature issupported as of 4.2(6), andallows VPC peers tosynchronize their ARPtables over CFS

vpc(config)# vpc domain 1vpc(config-vpc-domain)# ip arp synchronize

More information


75/124


More information

sh log last

(review sequence of events) show file logflash://sup-standby/log/messages

(in case other supervisor was active when everything started)

sh process log(which processes have crashed when)

sh redundancy status(status of supervisor redundancy & last switchover data)

sh system reset-reason(last reset/switchover reason per module)

sh logging onboard internal reset-reason(reset reason from different components point of view – usefulfor complex cases)

sh tech /from main VDC/(collects most of the above for offline analysis)


76/124


VPC Agenda

Initialization


Spanning Tree

Traffic forwarding

1st hop redundancy


Handling of Spanning Tree: VPC


77/124


Handling of Spanning Tree: VPC

STP process

Primary Secondary

STP process

STP runs on both switches (2 active control

planes) but only primary switch controls VPCs.(even if root is secondary , then Primary will send

bpdu with root info being secondary)

VPC port states changes are communicated to

secondary via CFS messages.

For non-VPC ports domain appears as 2 bridges

1

Peer-link is part of STP. BPDU handling ismodified such that Peer-link will never be blocked

(similar to MST implementation of IST)

2

Non-VPC ports are managed independently by

local STP process on each switch

1 1

2


78/124


STP troubleshooting

Peer link is running STP

Left-Root# sh spanning vlan 35

VLAN0035

Spanning tree enabled protocol rstpRoot ID Priority 24611

Address 001b.54c2.4241

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 24611 (priority 24576 sys-id-ext 35)

Address 001b.54c2.4241

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- --------------------------------

Po1 Desg FWD 1 128.4096 (vPC) P2p

Po100 Desg FWD 2 128.4195 (vPC peer-link) Network P2p

Right# sh spanning-tree vl 35 detail | i "^ Port|BPDU"Port 4096 (port-channel1, vPC) of VLAN0035 is designated forwarding


Port 4195 (port-channel100, vPC Peer-link) of VLAN0035 is root forwarding


On the other end of peer-link po1 is designated despite not sending or

receiving single BPDU

STP troubleshooting

This output can be easily limited to


79/124


g Looking at BPDUs

Left-Root# debug spanning-tree bpdu_tx tree 101

14:20:37.556707 stp: RSTP(101): transmitting RSTP BPDU on port-channel10014:20:37.556750 stp: vb_vlan_shim_send_bpdu(1933): VDC 4 Vlan 101 port port-

channel100 enc_type 1 len 42

14:20:37.556834 stp: RSTP(101): transmitting RSTP BPDU on port-channel1

14:20:37.556863 stp: vb_vlan_shim_send_bpdu(1933): VDC 4 Vlan 101 port port-channel1enc_type 2 len 36

Left-Root# debug spanning-tree all

14:22:23.560147 stp: RSTP(1): transmitting RSTP BPDU on port-channel100

14:22:23.560169 stp: vb_vlan_shim_send_bpdu(1933): VDC 4 Vlan 1 port port-channel100enc_type 2 len 36

14:22:23.560219 stp: BPDU TX: vb 1 vlan 1 port port-channel100 len 36 ->0180c2000000CFG P:0000 V:02 T:02 F:78 R:80:01:00:1b:54:c2:42:43 00000002B:80:01:00:1b:54:c2:42:44 9063 A:0000 M:0014 H:0002 F:000f

Left-Root# sh spanning-tree internal event-history tree 0 interface port-channel 50

VDC02 MST0000

0) Transition at 497772 usecs after Tue Oct 20 17:42:01 2009State: FWD Role: Root Age: 5 Inc: no [STP_PORT_STATE_CHANGE]

1) Transition at 661395 usecs after Tue Oct 20 17:42:01 2009

State: FWD Role: Root Age: 4 Inc: no [STP_PORT_ROLE_CHANGE]

2) Transition at 17741 usecs after Tue Oct 20 17:42:03 2009

State: BLK Role: Root Age: 5 Inc: no [STP_PORT_STATE_CHANGE]

...

Looking at past events…

p y

necessary Vlan/Interface, but it

doesn’t dump the BPDU

Very chatty – use ‘debug logfile

’ to redirect output to a file


80/124


STP inconsistencies

STP-2-VPC_PEER_LINK_INCONSIST_BLOCK: vPC peer-link detected BPDU receive timeoutblocking port-channel11 VLAN0121.

When STP detects certain abnormal situations it maymark ports as inconsistent and block them to preventforwarding loops

Root – Root Guard feature detected inconsistency(unwanted bridge tries to become root)

Loop – Loop Guard feature detected inconsistency

(port becomes designated because no BPDUs are beingreceived)

Bridge Assurance (BA)

(no BPDUs are received from remote side)

VPC Peer-link

(any of above inconsistencies happened on VPC peer-link)

Handling Peer-Link STP inconsistencies


81/124


gon Primary switch

Primary SecondaryWhen peer-link STP inconsistency is detected on

primary switch the link will be put in ‘inconsistent’

STP state (effectively blocking state)

1

BPDUs are not sent on peer-link when it is

inconsistent. This is to allow secondary switch to

detect inconsistency and react

1

i n c o n s i s

t e n c y

Handling Peer-Link STP inconsistencies


82/124


gon Secondary switch

Primary Secondary

When peer-link STP inconsistency is detected on

secondary switch the peer link will be put in

‘inconsistent’ STP state (effectively blockingstate)

1

Respective vlans or MST instances are also

blocked on all VPCs

22

2

1 i n c o n s i s

t e n c y

i n c o n s i s

t e n c y


83/124


Bridge assurance, Dispute & UDLD

BA is default enabled on Peer-Link (and recommended to remain

enable), not recommended for VPCs unless Peer-Switch feature isused

Dispute is default enabled (for both RSTP and MST on VPC)

UDLD [normal mode] is recommended to take out bad links fromchannels (otherwise LACP takes ~100sec vs ~20 with UDLD)

Recommendation

Preferred BA + UDLD + Dispute (on all interswitch links when usingPeer-switch) when all switches support this (nexus7000/5000 andcat6500/VSS do support)

Without Peer-switch BA should be kept only on Peer-Link (noBA/Loop guard on VPCs)

If preferred config is not supported use Loop Guard + UDLD(supported by all Cisco switches)

STP behavior upon VPC primary failure


84/124


p p y

Primary SecondaryOP-Primary

ROOT ROOTBackup

ROOT

Depending on control plane load it might take few

seconds for Op-primary to start sending BPDUs.

This might cause STP reconvergence on

connected switches hence increasing hello timeor peer-switch feature might be considered in

large deployments

Primary switch (STP root) fails1

Secondary switch becomes operational primary

and STP root

2

STP root port doesn’t change for access switch

nor any STP port states for VPCs, forwardingcontinues1

2

STP behavior upon VPC primary recovery


85/124


p p y y

SecondaryOP-Primary

ROOT ROOT

OP-Secondary

SYNC BackupROOT

Left switch comes back up1

Peer-Link comes back up2

VPC role is resolved as Operational-secondary3

Left switch has better STP priority becomesSTP root4

STP root port of right switch will change and that

will trigger SYNC: all non-edge STP ports will be

temporarily blocked

5

Once sync is complete ports will resume

forwarding

1

23

4 5

VPC Peer-Switch feature


86/124


Primary Secondary

Both VPC switches originate BPDUs with preconfigured information. Thisallows to keep the same BPDU when primary fails/recovers no extraSYNC required avoid short interruption in forwarding described on

previous slide is avoided

Both left and right switches consider themselves root

Both left and right switches send BPDUs all the time no need to raisehello time

Available 4.2(6) – 5.x software

spanning-tree vlan 1-1000 priority 8192vpc domain 1peer-switch

spanning-tree vlan 1-1000 priority 8192vpc domain 1peer-switch

ROOT ROOT

VPC Peer-Switch feature


87/124


Primary Secondaryleft# sh span vlan 101

VLAN0101Spanning tree enabled protocol rstp

Root ID Priority 8293Address 0023.04ee.be01This bridge is the root

...

Bridge ID Priority 8293 (priority 8192)Address 0023.04ee.be01

...

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- ---------------Po1 Desg FWD 1 128.4096 (vPC) P2pPo100 Root FWD 2 128.4195 (vPC peer-link)

left# sh vpc role | i macvPC system-mac : 00:23:04:ee:be:01 vPC local system-mac : 00:1b:54:c2:42:43

right# sh span vlan 101

VLAN0101Spanning tree enabled protocol rstpRoot ID Priority 8293

Address 0023.04ee.be01

This bridge is the root

...

Bridge ID Priority 8293 (priority 8192)Address 0023.04ee.be01

...Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- ---------------Po1 Desg FWD 1 128.4096 (vPC) P2pPo100 Desg FWD 2 128.4195 (vPC peer-link)

In Peer-Switch mode bridge-IDcomes from system-mac asopposed to local mac in normalmode

ROOT ROOT

M i f i


88/124


More information

show spanning-tree internal event-history all(allows to look back at past STP events, not included in sh tech)

sh tech stp(from both sides of VPC)

sh tech

(from both sides of VPC, this will include in it ‘sh tech stp’, in caseVPC is is non-default VDC collect also sh tech from VDC 1)

VPC A d


89/124


VPC Agenda

Initialization


Spanning Tree

Traffic forwarding

1st hop redundancy


Special case for forwarding


90/124


p g

x

x

x

PC A ends a packet to PC B1

MAC B is not known by left switch flood2

MAC B is not known by right switch flood3

B receives duplicate frames4

MAC A will be learned on wrong port on the lower

access switch blackholing traffic to A

5

Frames received on Peer-Link may not be flooded

out of VPCs

PC A

PC B

A ←

1

2 3

4

5 A ↑ x

Special case for forwarding:VPC i l t ti


91/124


VPC implementation

MAC B is not known by left switch flood1

Frames received from Peer-Link are never sent

out of VPC (except those without operational

ports on ingress switch)

Egress port ASICs will drop the frame

Frame is still flooded to devices that are solely

connected to egress switch3

This rule (called ‘VPC check’) stands for all traffic

(L2, L3, unicast, multicast, broadcast, flooded etc)

1

3

2

2

2

PC A

PC B

Summary: VPC traffic forwarding


92/124


√ √ X √

x

VPC f di d L3 i li ti


93/124


vPC view Layer 2 topology Layer 3 topology

Port-channel looks likea single L2 pipe.

Hashing will decidewhich link to chose

Layer 3 will use ECMP

for northbound traffic

7k1 7k2

R

7k1 7k2

R

7k vPC

R

R could be any router,

L3 switch or VSSbuilding a port-channel

VPC forwarding and L3 implication

R can Decide to send to 7k1 at L3 (next-hop = 7k1 if Po) and

uses link to 7k2 at L2 level !!!

Path is R 7k2

7k1 DROPPED (per VPC check) as

incoming on peer-link if it must be routed to another VPC

Layer 3 and vPC Design update


94/124

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKCRS-1930 96Router

7k1 7k2

Switch

Po1

Po2

Use L3 links to hook up routers and peer with a vPC domain

Don’t use L2 port channel to attach routers to a vPC domain unless you statically route toHSRP address

If both, routed and bridged traffic is required, use individual L3 links for routed traffic and L2port-channel for bridged traffic

Use of peer-gateway does NOT change above recommendations

Router

Switch

L3 ECMP

Po2

Layer 3 and vPC Design update

PP

P

Routing Protocol Peer

Dynamic Peering Relationship

P

P

L 3 d VPC id ti


95/124


Layer 3 and VPC – consideration

Best : use Routed links from VPC pair to routers

Alternative : VPC in a pure L2 VDC and routing in aseparate VDC

Do not make L3 routing protocol peering between

VPC pair of switches on a VPC vlan.May lead to routing frame towards Peer-link leading to dropper VPC-Check

If peering between VPC devices is needed, must be doneoutside of the peer link

Keep SVI interface administrative status in sync(both up or both down) – This is a type 2consistency check

Special case for L2 learning


96/124


A ↓ A

x

A ↓

MAC A is learned on lower VPC1

PC A

PC BMAC A is learned on Peer-Link2

Frame destined to A arriving to right switch will be

sent to Peer-Link

3

Traffic should prefer local links when available

(traffic locality rule)

1

2

3

L2 learning: VPC implementation


97/124


A ↓ A ↓

MAC A is learned on lower VPC1

PC A

PC B

MAC addresses are never learned from traffic on

Peer-Link

Frame destined to A arriving to right switch will be

sent out of lower VPC3

1

2

3

Left switch sends a CFS message to right switch

telling about MAC A learned on lower VPC. Right

switch updates MAC address table

2

CFS message

TroubleshootingPo50

Vlan 50

Po22

Vlan 20


98/124


gLayer 2

20.1.2.391.0.0.10

0013.1908.e246

nexus# sh mac address-table address 0013.1908.e246 vlan 50

VLAN MAC Address Type age Secure NTFY Ports---------+-----------------+--------+---------+------+----+------------------* 50 0013.1908.e246 dynamic 0 F F Po50

nexus# sh spanning-tree vlan 50 interface port-channel 50Mst Instance Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- --------------------------------MST0002 Desg FWD 200 128.4145 (vPC) P2p

nexus# sh hardware mac address-table 2 address 0013.1908.e246 vlan 50

Valid| PI | BD | MAC | Index | Stat| SW | Modi| Age | Tmr || | | | | ic | | fied| Byte| Sel |

-----+----+-------+---------------+--------+-----+----+-----+-----+-----+1 1 161 0013.1908.e246 0x00a36 0 3 0 141 1

nexus# sh system internal pixm info ltl 0x00a36 | i Eth.*,0x0a36 Eth2/36,

nexus# sh mac address-table address 0021.55e0.66c2 vlan 20

VLAN MAC Address Type age Secure NTFY Ports---------+-----------------+--------+---------+------+----+------------------* 20 0021.55e0.66c2 dynamic 660 F F Po22

nexus# sh spanning-tree vlan 20 interface port-channel 22Mst Instance Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- --------------------------------MST0000 Desg FWD 200 128.4117 (vPC) Network P2p

nexus# sh hardware mac address-table 1 address 0021.55e0.66c2 vlan 20Valid| PI | BD | MAC | Index | Stat| SW | Modi| Age | Tmr |

| | | | | ic | | fied| Byte| Sel |-----+----+-------+---------------+--------+-----+----+-----+-----+-----+1 1 18 0021.55e0.66c2 0x00a32 0 2 0 103 1

nexus# sh system internal pixm info ltl 0x00a32 | i Eth.*,

0x0a32 Eth1/13, Eth1/14,

MAC addresses should point

to expected ports in expected

vlans (path towards source)

The ports should be in STP

forwarding mode

Hardware MAC address

table should be consistentwith software table

Finding port# for given index

VPC

TroubleshootingLayer 3

Po50

Vlan 50

Po22

Vlan 20


99/124


Layer 3

nexus# sh routing ip 20.1.2.3...20.1.2.3/32, ubest/mbest: 1/0

*via 20.1.1.240, Vlan20, [1/0], 03:48:59, static

nexus# sh ip arp 20.1.1.240Address Age MAC Address Interface20.1.1.240 00:02:17 0021.55e0.66c2 Vlan20

nexus# sh forwarding ip route 20.1.2.3 module 2...------------------+------------------+---------------------Prefix | Next-hop | Interface------------------+------------------+---------------------20.1.2.3/32 20.1.1.240 Vlan20

nexus# sh forwarding adjacency 20.1.1.240 module 2

IPv4 adjacency information

next-hop rewrite info interface-------------- --------------- -------------20.1.1.240 0021.55e0.66c2 Vlan20

nexus# sh int vl 20 | i addressHardware is EtherSVI, address is 0023.ac66.1a42

nexus# sh mac address-table address 0023.ac66.1a42 vlan 20

VLAN MAC Address Type age Secure NTFY Ports---------+-----------------+--------+---------+------+----+------------------

G 20 0023.ac66.1a42 static - F F sup-eth1(R)

Is there route to

destination

Is the next hop resolved

Looking at module 2

because this is wherepackets in question

should be received

Is adjacency consistent

with ARP

Router MAC must have

Gateway flag in order for

packet to be L3 switched

20.1.2.391.0.0.10

0013.1908.e246

VPC

Where given packet will be load-balanced


100/124


For equal-cost routes

nexus# sh routing hash 91.0.0.10 20.1.2.3

Load-share parameters used for software forwarding:load-share mode: address source-destination port source-destinationUniversal-id seed: 0xcdb5769fHash for VRF "default"Hashing to path *20.1.1.3 (hash: 0x2a), for route:

20.1.2.3/32, ubest/mbest: 2/0*via 20.1.1.3, Vlan20, [1/0], 00:01:37, static*via 20.1.1.240, Vlan20, [1/0], 16:32:42, static

For port-channels

nexus# sh port-channel load-balance forwarding-path interface port-channel 22 dst-ip20.1.2.3 src-ip 91.0.0.10 vlan 20 module 2

Missing params will be substituted by 0's.

Module 2: Load-balance Algorithm: source-dest-ip-vlan

RBH: 0 Outgoing port id: Ethernet1/14

Load-balancing is configurable

under ‘ip load-sharing address’ in

default VDC and affects all VDCs

Load-balancing is configurable

under ‘port-channel load-balance’

in default VDC and affects all VDCs

Use ‘sh port-channel rbh-distribution’ to see which link sends traffic forwhich of 8 available load-balancing ‘buckets’

Hardware path packet drops

#1 command to look for hardware


101/124


nexus# sh hardware internal errors all----------------------------------------Hardware errors as reported in module 1----------------------------------------

|------------------------------------------------------------------------|

| Device:R2D2 Role:MAC ||------------------------------------------------------------------------|Instance:7ID Name Value Ports-- ---- ----- -----28688 aric_no_port_select_error 0000000000000002 1,3,5,7 I2...|------------------------------------------------------------------------|| Device:Ashburton Role:MAC Mod: 1 ||------------------------------------------------------------------------|Instance:03629 Egr

VSS VS VPC.pdf

Documents

Transcript of VSS VS VPC.pdf