Nexus 7000 and Nexus 3000 TAC Time - Jive Software · © 2012 Cisco and/or its affiliates. All...

© 2012 Cisco and/or its affiliates. All rights reserved.

Nexus 7000 and Nexus 3000"TAC Time"

Mike Pavlovich

Yogesh Ramdoss

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

• Introduction

• Nexus 7000 vPC Common Questions

• Nexus 7000 CoPP

• Nexus 7000 Ethanalyzer

• Nexus 7000 Handy Features

• Nexus 7000 Important Caveats

• Nexus 7000 References

• Nexus 3000 Basic Information

• Nexus 3000 Important Caveats / Hot Issues

• Nexus 3000 Best Practices

• Nexus 3000 References


• Mike Pavlovich

• CCIE R&S # 4284

• Technical Leader, Data Center Networking Team, Cisco Services

• Supports Nexus 7000, Nexus 6000, Nexus 3000, and Catalyst Switches.

• Yogesh Ramdoss

• CCIE R&S # 16183, VCP 5.0

• Technical Leader, Data Center Networking Team, Cisco Services

• Supports Nexus 7000, Nexus 6000, Nexus 3000, and Catalyst Switches.


• Nexus 7000 vPC Common Questions • Nexus 7000 CoPP• Nexus 7000 Ethanalyzer• Nexus 7000 Handy Features • Nexus 7000 Important Caveats• Nexus 7000 References


Why do we need vPC Auto-Recovery?• Scenario 1: Power outage shuts down both Nexus 7000 vPC peers

simultaneously and only one switch is able to come back up (5.0(2) “reload restore” feature addressed this but not scenario 2 below)

• Scenario 2: vPC peer-link is lost first and then the primary vPC peer switch is powered down

Configuration of vPC auto-recovery (5.2(1)):

• vPC auto-recovery timeout: to see if either the vPC peer-link comes up or peer-keepalive status is up. If so then auto-recovery will not get triggered. Default 240 sec – configurable via “auto-recovery reload-delay x” (x = 240-3600 sec)

S1 (config)# vpc domain 1S1 (config-vpc-domain)# auto-recovery

S2 (config)# vpc domain 1S2 (config-vpc-domain)# auto-recovery


Scenario 1

0


Scenario 2

0


• Q: Which Nexus 7000 would respond to a ping or ARP request for the Virtual IP Address (HSRP/VRRP) in a vPCscenario?

• A: The HSRP Active will respond (when both vPC legs are up)

• vPC is Active/Active for HSRP/VRRP regarding L3 switching of traffic received with the destination virtual mac address. Both peer switches own the virtual MAC so which ever peer gets the packet will forward it and not send it across the peer link

• The virtual IP address on the other hand is owned by the HSRP/VRRP active Nexus 7000 in a vPC scenario


Did you know?...

• The CoPP rates are applied per FE (forwarding engine) on the line cards. Traffic hitting the CPU = conform rate x # of FE’s

• F1 modules do not use CoPP. They use hardware rate-limiters instead

Nexus7K(config)# hardware rate-limiter f1 ? rl-1 STP and Fabricpath-ISIS rl-2 L3-ISIS and OTV-ISIS rl-3 UDLD, LACP, CDP and LLDP rl-4 Q-in-Q and ARP request rl-5 IGMP, NTP, DHCP-Snoop, Port-Security and Mgmt traffic

M1 modules = 1 x FE (2 for M108)M2 modules = 2 x FEF2 modules = 12 x FE (SoC = Switch on Chip)


• Four options for default policies:

• All of these options use the same class-maps and classes, but different rate & burst values

• CoPP is configured on default VDC but effects all VDC’s. CoPP is applied per FE so recommend all ports on same FE be part of same VDC if possible

• Prior to 5.2(1), the setup command was used to change the CoPP option. 5.2(1) introduced the copp profile command.

StrictModerateLenientDense (introduced in 6.0(1))

Applied if no option is selected or if set up is skipped

recommended if majority of modulesare F2 Series (12 FE’s each)


Nexus7k# conf tNexus7k(config)# copp profile strictNexus7k(config)# exit

Nexus7k# show copp statusLast Config Operation: copp profile strictLast Config Operation Timestamp: 20:40:27 PST Apr 21 2013Last Config Operation Status: SuccessPolicy-map attached to the control-plane: copp-system-p-policy-strict

Nexus7k# show copp diff profile strict profile moderate<output left out to save space… but you get the idea>

Nexus7k# conf tNexus7k(config)# control-planeNexus7k(config-cp)# scale-factor 2.0 module 1Nexus7k# show system internal copp info<snip>Linecard Configuration:-----------------------Scale FactorsModule 1: 2.00etc…

Scale-factor-6.0 feature to increase or reduces the policer rate of the applied CoPP policy for a particular linecard-Value ranges from 0.10 to 2.0-Recommended for chassis with M & F2 modules


Nexus7k(config)# policy-map type control-plane copp-system-policy-strictNexus7k(config-pmap)# class copp-system-class-criticalNexus7K (config-pmap-c)# logging drop threshold 10000 level 5

Sample Syslog:

10000 = # of bytes

%COPP-5-COPP_DROPS5: CoPP drops exceed threshold in class: copp-system-class-critical, check show policy-map interface control-plane for more info.

Nexus7k# show policy-map interface control-plane | i"class|conformed|violated|module"

class-map copp-system-class-critical (match-any)module 1:conformed 123126534 bytes; action: transmitviolated 143021 bytes; action: drop

module 2:etc..

Nexus7K# show hardware rate-limiter | in "Module|f1|Class”Module: 1 R-L Class Config Allowed Dropped Total f1 rl-1 4500 0 f1 rl-2 1000 0etc…


• Capture traffic to/from CPU associated with module interfaces (inband) or Sup MGMT interface (mgmt)

Nexus7k# ethanalyzer local interface ?inband Inband/Outband interfacemgmt Management interface

Nexus7k# ethanalyzer local interface inband ?<CR> > Redirect it to a file>> Redirect it to a file in append modeautostop Capture autostop conditioncapture-filter Filter on ethanalyzer capturecapture-ring-buffer Capture ring buffer optiondecode-internal Include internal system header decodingdetail Display detailed protocol information display-filter Display filter on frames capturedlimit-captured-frames Maximum number of frames to be captured (default is 10)limit-frame-size Capture only a subset of a frameraw Hex/Ascii dump the packet with possibly one line summarywrite Filename to save capture to| Pipe command output to filter

Nexus7k#


Nexus7k# ethanalyzer local interface inbandCapturing on inband2013-04-15 17:20:53.864238 c8:9c:1d:39:87:34 -> 01:80:c2:00:00:0e LLC U, func=UI; SNAP, OUI 0x00000C (Cisco), PID 0x01342013-04-15 17:20:53.922775 00:23:33:74:47:04 -> 01:80:c2:00:00:00 STP Conf. Root = 32768/1/00:23:33:74:47:00 Cost = 0 Port = 0x80052013-04-15 17:20:53.977277 00:1b:54:c1:73:53 -> 01:00:0c:cc:cc:cd STP RST. Root = 32768/95/00:24:98:6f:ba:c3 Cost = 0 Port = 0x905e2013-04-15 17:20:53.985859 00:15:fa:42:5d:98 -> 01:80:c2:00:00:00 STP MST. Root = 4096/0/00:13:5f:20:bb:80 Cost = 0 Port = 0x96862013-04-15 17:20:53.986011 00:01:00:01:00:01 -> 01:80:c2:00:00:0e LLC U, func=UI; SNAP, OUI 0x00000C (Cisco), PID 0x88402013-04-15 17:20:54.278543 70:ca:9b:95:cc:a5 -> 01:80:c2:00:00:41 0x22f4 Ethernet II2013-04-15 17:20:54.396876 f8:66:f2:e4:b9:dd -> 01:80:c2:00:00:41 0x888a Ethernet II2013-04-15 17:20:54.476706 10.10.10.2 -> 10.10.10.1 UDP Source port: 3200 Destination port: 32002013-04-15 17:20:54.515927 10.10.16.6 -> 224.0.0.10 EIGRP Hello2013-04-15 17:20:54.516058 10.10.16.6 -> 224.0.0.10 EIGRP Hello

10 packets capturedNexus7k#

Note: <cntl>C will stop the ethanalyzer captureNote: by default the output is displayed on your screen. To save the output to a file use the “write” & “read” options

Nexus7k# ethanalyzer local interface inband capture-filter "stp" limit-captured-frames 2Capturing on inband2013-04-15 17:12:42.289309 00:15:fa:42:5d:98 -> 01:80:c2:00:00:00 STP MST. Root = 4096/0/00:13:5f:20:bb:80 Cost = 0 Port = 0x96862013-04-15 17:12:42.616792 88:43:e1:c7:4d:b8 -> 01:80:c2:00:00:00 STP MST. Root = 4096/0/00:13:5f:20:bb:80 Cost = 200 Port = 0x9000

2 packets capturedNexus7k#

Nexus7k# ethanalyzer local interface inband capture-filter "host 10.10.16.6" limit-captured-frames 1 write bootflash:testCapturing on inband1 Nexus7k# ethanalyzer local read bootflash:test2013-04-15 17:29:15.679219 10.10.16.6 -> 224.0.0.10 EIGRP HelloNexus7k#


Capture Filter Traffic Captured

host 1.1.1.1 to or from a host

net 172.16.7.0/24 (or “net 172.16.7.0 mask 255.255.255.0”) to or from a range of IP addresses

src net 172.16.7.0/24 (or “src net 172.16.7.0 mask 255.255.255.0”) from a range of IP addresses

dst net 172.16.7.0/24 (or “dst net 172.16.7.0 mask 255.255.255.0”) to a range of IP addresses

port 53 only certain protocol e.g. DNS

port 67 or port 68 DHCP traffic

host 172.16.7.3 and not port 80 and not port 25 is not certain protocols E.g. not HTTP or SMTP

port not 53 and not arp except ARP & DNS

ip only IP traffic

not broadcast and not multicast only unicast traffictcp portrange 1501-1549 within a range of Layer 4 portsether proto 0x888e based on Ethernet type E.g. EAPOLether proto 0x86dd IPv6 captureip proto 89 IP protocol typenot ether dst 01:80:c2:00:00:0e Reject Ethernet frames based on mac address


Nexus7k# checkpointProcessing the Request... Please Wait................................. DoneNexus7k# show checkpoint summaryUser Checkpoint Summary--------------------------------------------------------------------------------1) user-checkpoint-1:Created by adminCreated at Fri, 06:49:06 26 Apr 2013Size is 39,156 bytesDescription: None

Nexus7k# config tNexus7k(config)# no vlan 20, 30

Nexus7k# rollback running-config ?checkpoint Rollback running configuration to checkpointfile Rollback running configuration to configuration file

Nexus7k# rollback running-configuration checkpoint user-checkpoint-1Note: Applying config parallelly may fail Rollback verificationCollecting Running-ConfigGenerating Rollback PatchExecuting Rollback PatchGenerating Running-config for verificationGenerating Patch for verification

Nexsu7k# clear checkpoint databaseProcessing the Request... Please Wait.................................. Done

Caution! Clears all saved configurationsClear Checkpoint Database:

Automatically puts VLAN 20 & 30 back into running Configuration

Rollback to configuration “user-checkpoint-1”

Rollback Configuration:

Create and Verify Checkpoint Configurations:Create checkpoint configuration (default name = “user-checkpoint-#”)You can create up to 10 configurations per VDC

Manually Removed VLAN 20 & 30


• GZIP

• Accounting Log

Nexus7K# show tech-support gold > bootflash:tech_goldNexus7K# dir bootflash:tech_gold5944217 Apr 26 09:14:13 2013 tech_gold

Nexus7K# gzip bootflash:tech_goldNexus7K# dir bootflash:tech_gold.gz332467 Apr 26 09:14:13 2013 tech_gold.gz

Nexus7K# config tNexus7K(config)# feature ospfNexus7K(config)# exitNexus7K# show accounting log | i “Apr 26”Fri Apr 26 09:24:18 2013:type=update:id=console0:user=admin:cmd=configure terminal ; feature ospf (SUCCESS)Nexus7K#


• Locator LED / Beacon

• Show {blah} | no-more

Nexus7K# locator-led ? chassis Blink chassis led fan Blink Fan led module Blink module led powersupply Blink powersupply led xbar Xbar

Nexus7K# locator-led chassisNexus7K# show locator-led status

Nexus7K# config tNexus7K(config)# interface ethernet 1/1Nexus7K(config-if)# beacon

Nexus7K# show running-config | no-more

Output of given show command will complete without the need to hit the space bar


• Show {blah} | diff

• Send Message Nexus7K# send ?

LINE Send message (a line) to all open sessions session Send message to specific session

Nexus7K# send We found the problem, its not the NexusBroadcast message from admin (Fri Apr 26 10:06:27 2013):We found the problem, its not the Nexus

Nexus7K#

Nexus7K# show interface ethernet 1/1 | diff2,3c2,4< admin state is up, Dedicated Interface< Hardware: 10000 Ethernet, address: d0d0.fd9d.a680 (bia d0d0.fd9d.a680)---> admin state is down, Dedicated Interface> Hardware: 10000 Ethernet, address: 0024.f714.3541 (bia d0d0.fd9d.a680)> Internet Address is 172.10.10.1/24


• Only effects F2 module admin down ports (Non-impacting)

• ISSU from 6.0(x) to 6.0(x) and then ISSU to 6.1(x) is trigger

• ISSU from 6.0(x) direct to 6.1(x) will not see the issue

• Workaround: Reload module

• Resolved: in 6.1(1) and later

• DDTS: CSCua03125

%DIAG_PORT_LB-2-PORTLOOPBACK_TEST_FAIL: Module:7 Test:PortLoopback failed 10 consecutive times. Faulty module: affected ports:1,3,4,10-12,14-17,20,25,27,28,37,39-41,43-46,48 Error:TestFailed, Could not identify the Faulty Device

show diagnostic result module {x} detail 6) PortLoopback:

Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 -----------------------------------------------------

U U U U . . . . . . . . E E U U


• Some packets in an L2 vlan (No SVI configured) are still hitting CoPP although they are not sent to the CPU

• Limited to packets with unicast IP address and a multicast/broadcast mac address. This includes ARP and DHCP requests

• These packets may congest the policy and could cause other traffic hitting the same policy to be dropped

• Day 1 behavior (all software releases)

• Workaround: None

• Resolved: in progress

• DDTS: CSCub47533


• 6KW AC power supplies that have SN starting with “AZS” may inadvertently shutdown momentarily when power is restored to input 1 after a power failure of two or more supplies

• Only occurs on input 1, input 2 does not exhibit the problem

• 6KW AC power supplies starting with “DTM” are not effected

• This has been seen in power grid redundancy testing when the power grid feeding input 1 of two or more power supplies is shut down so that the power supplies run off of power from input 2 only

• When the power grid for input 1 is restored again a power supply might shut down momentarily and then recover

• Resolution: Replace the existing PS unit

• DDTS: CSCtt38629


• This can take down the entire data center for if all of the Nexus 7000’s were reloaded at the same time. This may come into effect after approximately 3 months of uptime of an active supervisor

• Irrespective of any feature turned on/off

• Workaround: Reload of the active supervisor will clear the issue in a setup with two supervisor cards. Reload of the switch will clear the issue in a setup with a single supervisor

• Resolved: 5.1(4) or 5.2(1) or later

• DDTS: CSCtq62339

%PLATFORM-2-MEMORY_ALERT: Memory Status Alert: MINOR. Usage 85% of Available Memory%PLATFORM-2-MEMORY_ALERT: Memory Status Alert: SEVERE. Usage 90% of Available Memory


• ISSU upgrade from earlier image to 5.2(1)-(6) then ISSU to 5.2(7) or later

• This may result in degraded performance with the following error messages and possibly an ipfib process crash

• Workaround1: Configure the following regardless if LISP is in use

• Workaround2: If the issue is already hit, reload the effected modules

• DDTS: CSCub96980

%IPFIB-SLOT2-2-FIB_TCAM_HA_ERROR: FIB recovery errors, please capture 'show tech forwarding l3 unicast' and 'show tech forwarding l3 multicast'

feature lispConfigure "ip lisp etr" for all vrfs followed by "no ip lisp etr"no feature lisp


• Nexus 7000 Product Page on CCO:http://www.cisco.com/en/US/products/ps9402/index.html

• Nexus 7000 White Papers:http://www.cisco.com/en/US/products/ps9402/prod_white_papers_list.html

• Nexus 7000 Data Sheets:http://www.cisco.com/en/US/products/ps9402/products_data_sheets_list.htm

• Nexus 7000 Presentations: http://www.cisco.com/en/US/products/ps9402/prod_presentation_list.html

• Nexus 7000 Recommended NX-OS:http://www.cisco.com/en/US/docs/switches/datacenter/sw/nx-os/recommended_releases/recommended_nx-os_releases.html

• Nexus 7000 Scalability Guide:http://www.cisco.com/en/US/docs/switches/datacenter/sw/verified_scalability/_Cisco_Nexus_7000_Series_NX-OS_Verified_Scalability_Guide.html


• Nexus 3000 Basic Information• Nexus 3000 Important Caveats / Hot Issues• Nexus 3000 Best Practices• Nexus 3000 References


3548 3064X 3064T 3016Q 3048

Algorithm Boost Yes No No No No

Switch Capacity 960 Gbps 1.28 Tbps 1.28 Tbps 1.28Tbps 176 Gbps

Interface Type 48 SFP+ 28 SFP+ and 4 QSFP+

48 RJ45 and 4 QSFP+

16 QSFP+ 48 RJ45 and 4 SFP+

Max 1 GE ports 48 48 48 48 48

Max 10 GE ports 48 Up to 64 Up to 64 Up to 64 4

Max 40 GE ports 0 Up to 4 4 Up to 16 0

Switch Latency < 250 nsec < 1 usec 3-4 usec < 1 usec 2-8 usec

Line-rate on all ports (for L2 and L3 traffic)

Yes Yes Yes Yes Yes

Rack Unit 1 1 1 1 1

Hot-swappable PSU and Fan ?

Yes Yes Yes Yes Yes


3016/3048/3064 3064-T 3548


Layer 2802.1w, 802.1s, RPVST+, Root Guard, Uplink Guard, Bridge Assurance, PortFast, CDP, UDLD, PVLANs, IGMP Snooping, 802.1Q trunks, Port-

Channel, LACP, SVI, SPAN, Jumbo Frames, NTP

Management/Security

DHCP snooping, DAI, Radius, Tacacs+, AAA, CallHome, SSHv1/V2, telnet, IPv4 & IPv6 mgmt, SNMP MIBs, Traps, EthAnalyzer, RBAC, syslog, core dump, RMON, first-setup script, accounting log

System/Operations POST, OHMS, OBFL

ACL/QOSPACLs, VACLs, RACLs, Session based ACLs, ACL based QOS (CoS/DSCP marking), egress Bandwidth Limiting, 802.1p priority, strict priority scheduling, Tail Drop, ECN, WRED, Storm Control (broadcast, multicast)

Layer 3L3 Physical & SVI routed interfaces, static routing, RIP-v2, OSPF-v2, OSPF

fast convergence, EIGRP-IPv4, BGP, ECMP, IGMP v1/v2/v3, MSDP, PIM-v2 for IPv4, PIM-SSM for IPv4, HSRP, VRRP, VRF-lite, SPAN for L3 interfaces

Note: 5.0.3 do NOT support IPv6, OSPFv3, BFD and PBR


Feature Details

L3 Interfaces L3 Physical, SVI, Port‐Channel, Sub‐Interface

IPv4 Routing Protocol RIPv2,OSPF, EIGRP,BGPv4

Multicast PIM‐SM, SSM, MSDP, IGMP v1‐3, IGMP Snooping

HSRP/VRRP Yes

ECMP Yes (32‐way)

VRF Lite Yes

L3 SPAN Yes

uRPF – Strict & Loose mode Yes

Layer 2 CDP, UDLD, PVLANs, 802.1Q trunks, NTP, LACP

Spanning Tree & Extensions 802.1w, 802.1s, RPVST+, Root Guard, Loop Guard, BPDU Guard, Bridge Assurance, PortFast

PVLAN Trunks No


Feature Details

Traffic Storm Control Broadcast, Multicast, Unknown‐Unicast

System Management RBAC, Online Diag, SysLog, Call Home, SNMP, RMON, SPAN

Management Security AAA,RADIUS, TACACS+, SSHv1/v2, Telnet,IPv4/IPv6 Management

Security PACL, VACL, RACL, DHCP Snooping, DAI, IPSG, ACL on VTY

QOS Cos /DSCP Marking, Egress Bandwidth Limiting, Strict Priority Scheduling, WRR, WRED, ECN

DCB 802.1p, ETS

Jumbo Frames Yes (9216 Bytes)

MTU Per System


• Licensing scheme identical to Nexus 5000’s

• Customers must buy and install both licenses for full L3 support

Do “show license usage” to see the license in use.

Base (N3K-BAS1K9)Basic L3 features Inter‐VLAN routing, Static routes, RIPv2, ACLs, OSPFv2

(limited to 256 routes), EIGRP stub, HSRP, VRRP and uRPFIP Multicast PIM SM, SSM, MSDP

LAN Enterprise (N3K-LAN1K9)Advanced IPRouting

OSPFv2, EIGRP, BGP and VRF‐Lite

System Default (no PID)Comprehensive L2 feature set

vPC, VLAN, 802.1Q Trunking, LACP, UDLD (Std. and Aggressive), MSTP, RSTP, STP Guards, VTP Transparent

Security AAA, DHCP Snooping, Storm Control, PVLAN, CoPP

Management PTP, ERSPAN, DCNM support, Console, SSHv2 access, CDP, SNMP, Syslog


vPC peer—a vPC switch, one of a pair

vPC member port—one of a set of ports (port channels) that form a vPC

vPC—the combined port channel between the vPC peers and the downstream device

vPC peer-link—link used to synchronize state between vPC peer devices, must be 10GbEvPC

connected device

vPC member

port

vPCvPC

member port

vPC peer-link

non-vPC connected

device

vPC peer


vPC peer-keepalive link—the keepalive link between vPC peer devices

vPC VLAN—one of the VLANs carried over the peer-link and used to communicate via vPC with a peer device

non-vPC VLAN—One of the STP VLANs not carried over the peer-link

CFS—Cisco Fabric Services protocol, used for state synchronization and configuration validation between vPC peer devices

vPC Peer-keepalive link

CFS protocol


“show interface X” results indicate that “input discard” counter increments when PIM, EIGRP and other control-plane packets are received.

This counter increments as these packets are redirected to the CPU by an ACL entry, and stop forwarding them to other front-panel ports.

N3K# show int eth 1/1Ethernet1/1 is up<snip>

RX5 unicast packets 3714544619 multicast packets 0 broadcast packets3714544625 input packets 475461709100 bytes<snip>0 input with dribble 3714544619 input discard

No workaround available. Issue is resolved in 5.0(3)U2(1) and later releases.

Ref bug ID: CSCto53539


“show hardware internal indiscard-stats front-port X” results shows all the counters as zero, which is incorrect.

“show interface X” shows non-zero indiscard counter valueNexus3000-1# show interface Ethernet 1/7Ethernet1/7 is up <output omitted>0 input with dribble 33844 input discard(includes ACL drops)<output omitted>N3K# show hardware internal interface indiscard-stats front-port 7+-----------------------------------------+-----------------+-----+| Counter Description | Count | Last Increment | +----------------------------------+-------------+----------------+IPv4 Discards 0 0STP Discards 0 0Policy Discards 0 0ACL Drops 0 0Receive Drops 0 0Vlan Discards 0 0+-----------------------------------+------------+----------------+

Issue is resolved in 5.0(3)U3(1) and later releases. Reference bug ID: CSCtu29771


In a High-Performance-Trading (HPT) setup, users may experience gap in the multicast streams.

The Nexus3000 switch receives lots of IGMP leaves and joins, and none of them dropped by CoPP (control-plane policing).

NX-OS release 5.0(3)U2(2) and later has improved performance.

Reference bug ID: CSCtt18984


After Nexus 3016 and 3064 is upgraded to 5.0(3)U3(1) releases, the switch may report:

%NOHMS-2-NOHMS_ENV_ERR_FAN_SPEED: System minor alarm in fan tray 1: fan speed is out of range on fan 4. 2495 to 12600 rpm expected. 2457 rpm read

%NOHMS-2-NOHMS_ENV_ERR_FAN_SPEED: System minor alarm in fan tray 1: fan speed is out of range on fan 8. 2200 to 12600 rpm expected. 1944 rpm read

This is surely not an hardware issue.

Issue is fixed in 5.0(3)U3(2) and later releases.

Reference bug ID: CSCty64730


Nexus3000 reports following messages when a parity error is detected:

%USER-3-SYSTEM_MSG: bcm_usd_isr_switch_event_cb:431: slot_num0, event 2, memory error type 0x1, mem addr 0x5f36, cause bit <addr> -bcm_usd

Switch needs to be reloaded to recover from this situation, otherwise the device may have functional impact.

Switch upgraded to 5.0(3)U5(1a) can detect and correct single bit parity errors.

Reference bug ID: CSCtw75636


Nexus3048 data sheet indicates that it supports up to 16K host entries.http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps11541/data_sheet_c78-685363.html

But, the “show hardware profile status” supports only 8K host entries.N3K# show hardware profile status Total LPM Entries = 16383 Total Host Entries = 8192Reserved LPM Entries = 1024 Max Host Limit Entries = 4096 Max Host6 Limit Entries = 0Max Mcast Limit Entries = 4000 Used LPM Entries (Total) = 3

This issue is reported in 5.0(3)U5(1b) and resolved in 5.0(3)U5(1c).

Reference bug ID: CSCug25153


Symptoms: Nexus3048 uplink ports are showing down.

This issue is seen when the ports are set to 1000 Mbps and Auto-negotiation enabled.

N3K# show running int e1/49interface Ethernet1/49speed 1000negotiate auto

N3K# show interface statusEth1/49 1 eth access down Link not connected

Issue is resolved in 5.0(3)U4(1) and later releases. As a workaround use “no negotiate auto” and “auto nonegotiate” commands, as applicable.

Reference big IDs: CSCty91237 and CSCtu68315


BFD hap resets intermittently.

This issue is seen when BFD sessions are configured over eBGP session.

Issue is resolved in 5.0(3)U2(2d) and later releases.

Reference bug ID: CSCts95614


For Nexus 3016, 3048, 3064 and 3064-T the recommended releases are 5.0(3)U5(1) or later, and minimum recommended release is 5.0(3)U4(1).

Please review the Release-notes before upgrading the NX-OS release: http://www.cisco.com/en/US/products/ps11541/prod_release_notes_list.html


N3k‐2N3k‐1

No parallel link between vPC peers – e.g., as an uplink backup Any parallel link set up should be:

A peer keep-alive link A trunk/channel carrying non-vPC vlans

No L2 / L3 links or port-channels parallel to vPC peer-link


No router(s) behind vPC

Switch

Unicast traffic works Multicast would not work with this topology, as do not sync PIM states

between vPC-peers. Same restriction applicable to IGP protocols.

N3K-1 N3K-2N3K-1 N3K-2


L2

L3

N3k‐1 N3k‐2

Router

L3L3

L2

L3N3k‐1 N3k‐2

Router

L2L2

L2

L3N3k‐1 N3k‐2

Router

VPC

✔

Not recommended / supported topology Multicast will have the issue with PIM over L3 vPC

No L3 vPC to uplink Router


How HSRP works ?

With vPC both HSRP active and standby can forward traffic.

The HSRP MAC is programmed in such a way that is L3 switched only if HSRP is in active/standby pair

Take Away is …

HSRP aggressive timers are not useful in a vPC topology

HSRP preempt-delay is not useful in a vPC topology

vPC

HSRP active HSRP standby

HSRP aggressive timers and preempt-delay are not useful


Have backup route over peer-link by having IGP peering

However, avoid unnecessary peering on all available vlans (over vPC peer-link), by using “passive” command

Have peering over only single or few VLAN

IGP peering

Have backup IGP path


vPCPrimary

vPCSecondary

L3L2

OSPF

N3K(config-vpc-domain)# delay restore ?<1-3600> Delay in bringing up vPC links (in seconds)N3K(config-vpc-domain)# delay restore 360

vPC interaction with Routing convergence on system restart

After a vPC device reloads and come back up routing protocols need time to reconverge

vPCs may black-hole routed traffic from access to core until layer 3 connectivity is reestablished

Tune vPC Delay Restore to avoid the traffic drop when device comes up

Fine-tune vPC delay-restore timer


N3k‐1 N3k‐2

PIM-RP

L3L3

L3 Link

L2 Link

PIM-RP

For better use of Peer-link bandwidth Better Convergence in case of failure due to pre-built SPT

vPC peers have equal cost to RP


When several hundreds of of servers joining multiple multicast groups at the same time (market opens), it generates a burst of IGMP traffic to N3K CPU/Control-Plane. Default CoPP rate-limiter for IGMP (400 pps) may drop to protect the CPU, and end-users may see delay in receiving data streams.

Also, due to the default MRT (10 sec) there is a possibility of IGMP traffic burst towards the CPU, when the network has several hundred hosts.

To handle the IGMP bursts ….

Increase the default maximum-response-time to 25 seconds

Increase the CoPP rate-limiter for IGMP to 600 pps

N3K(config)#interface vlan 101N3K(config-vlan)# ip igmmp query-max-response-time 25N3K(config-vlan)# ip igmp last-member-query-response-time 25

Increase CoPP limit and max-response-time (MRT) for IGMP

policy-map type control-plane copp-system-policyclass copp-s-igmp

police pps 600


Supported Topology


Nexus3000 Product Page on CCO: http://www.cisco.com/en/US/products/ps11541/index.html

Nexus3000 White Papers:http://www.cisco.com/en/US/products/ps11541/prod_white_papers_list.html

Nexus3000 Data Sheets: http://www.cisco.com/en/US/products/ps11541/products_data_sheets_list.html

Nexus3000 Presentations: http://www.cisco.com/en/US/products/ps11541/prod_presentation_list.html

Nexus3548 – Product Page and Algo Boost:http://www.cisco.com/en/US/products/ps12581/index.html

Understanding Nexus 3000 Switch Latency: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps11541/white_paper_c11-661939.html

Understanding “Input Discard” Interface Counter in Nexus3000:https://supportforums.cisco.com/docs/DOC-23994

Thank you.Thank you.

Nexus 7000 and Nexus 3000 TAC Time - Jive Software · © 2012 Cisco and/or its affiliates. All...

Documents

Transcript of Nexus 7000 and Nexus 3000 TAC Time - Jive Software · © 2012 Cisco and/or its affiliates. All...