Cisco CRS-3 Carrier Routing System MultiChassis...
Transcript of Cisco CRS-3 Carrier Routing System MultiChassis...
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Cisco CRS-3 Carrier Routing System
MultiChassis Overview Session ID – BRKARC-3002
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Agenda
CRS (Carrier Routing System) Overview
CRS-3 Overview
Overview of CRS Data Path
Overview of CRS Switch Fabric
CRS MultiChassis Specifics
CRS MultiChassis Switch Fabric details
CRS MultiChassis Control Ethernet
CRS MultiChassis Configuration
CRS MultiChassis Troubleshooting (Control Ethernet & Fabric)
3
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Agenda – continued Appendix for reference
Appendix Typical migration Step SC to MC
Appendix A Case Study 1 - Offline SC to MC migration
Appendix B Case Study 2 – Online SC to MC migration
Appendix C FCC Physical Installation notes
4
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS (Carrier Routing System) Overview
A fully modular and distributed routing system
CRS-1 (40G) and CRS-3 (140G) Systems
4,8 & 16 Slot Standalone Systems
8 & 16 Slot MultiChassis Systems (LCC‟s + FCC‟s)
5
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-1 & 3 Hardware Introduction
16 slot Line Card
Chassis with
integrated rack
(standalone or LCC)
External
Interfaces (PLIM)
Redundant
Route Processors
Fabric Card
Chassis (FCC)
8 slot Line Card
Chassis
rack mountable
Front and Rear
Access Required
MSC and Fabric
Access - Rear
6
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-3 Overview
New 140G Fabric and Line Cards
Increased throughput and scale
Backwards compatible with CRS-1 Hardware
4,8 & 16 Slot Standalone Systems
8 & 16 Slot MultiChassis Systems (LCC‟s + FCC‟s)
Can leverage 40G Line Card's & MSC‟s
7
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-3 Product Family
Foundation for the IP NGN Core
1.12 Tbps 2.24 Tbps 4.48 Tbps 4.48 Tbps to 322 Tbps
CRS-4/S CRS-8/S
CRS-16/S CRS-MC
Unprecedented service flexibility
Continuous system operation True Telco grade OS—IOS-XR
In-service hardware and software upgrades
Hitless upgrade to Multichassis
Unparalleled system longevity Multi-chassis fabric scales to 322Tbps.
Investment protection—common forwarding engines and I/O modules Modular hardware and software
8
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-3 Line Card Architecture
High Level LC Architecture
MSC Architecture
‒PSEs (Ingress and Egress)
‒IngressQ
‒FabricQ
‒EgressQ
Switch Fabric Architecture
Control Ethernet
9
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-3 System Overview
MAC 100G PHY
PLA
• Same architecture as CRS-1 but at 140G
• Compatible with all CRS-1 chassis sizes
• Standalone or MultiChassis
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3
S3
PLIM MSC/FP
Fabric
PSE
EgressQ
PSE
Intel CPU Sub-system
FabricQ
FabricQ
IngressQ
1 of 8 2 of 8
3 of 8 4 of 8
5 of 8 6 of 8
7 of 8 8 of 8
S1/S2/S3
S2
10
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-3 Line Card
from PLIM
120G 2x80G
PSE
160G
160G EgressQ
PSE
100G
100G
Intel CPU Sub-system
141G
113G
113G FabricQ
FabricQ
IngressQ
Output Queuing Output Features Multicast Replication Cell Reassembly
Forwarding Lookup Input Features
Queuing for Fabric Cell Segmentation Input Shaping
to PLIM
to Fabric
from Fabric
160G 2x100G
11
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-3 New hardware
CRS-3 New Hardware Boards Supporting Platforms
CRS-4 S3 Fabric Single Chassis CRS-4
CRS-8 S3 Fabric Single Chassis CRS-8
CRS-16 S3 Fabric Single Chassis CRS-16
CRS-16 S13 Fabric MultiChassis Line Card Chassis
FCC S2 Fabric Board MutiChassis Fabric Card Chassis
CRS-3 MSC 140G All CRS-3 platforms
CRS-3 14x10GE PLIM All CRS-3 platforms
CRS-3 20x10GE PLIM All CRS-3 platforms
CRS-3 1x100GE PLIM All CRS-3 platforms
Note: CRS-3 Fabric is backward compatible with CRS-1 40G MSC/PLIMs. Thus, a CRS-3 MultiChassis will support CRS-1 Line Card chassis. CRS-3 PLIMs are compatible only with CRS-3 MSCs and vice versa.
12
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-3 MSC140 vs. FP140
MSC140 FP140
Feature Without licenses With appropriate licenses
Queues 64K / slot 8 / port 8 / port
L3 Interfaces 12K / slot 250 / slot 250 / slot
Multichassis Yes No Yes
Netflow Sampling <1: 1500 >1: 1500 <1:1500
L2/L3VPN 2K+ VRFs/LC No VRFs 250 VRFs + L2VPN connections per LC
Route Scale 4M IPv4/2M IPv6 1M IPv4/500K IPv6 4M IPv4/2M IPv6
TE Scale >3K midpoints 3K midpoints + heads + tails >3K midpoints
Advanced Features Tunneling, LI No Tunneling or LI Tunneling, LI
13
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS System Attributes Comparison MSC40 CRS-1 MSC140 CRS-3
Bandwidth 40 Gbps 140 Gbps
Max Packet per second 80 Mpps 125 Mpps
FIB Scalability 2M IPV4, 1M IPV6 4M IPV4, 2M IPV6
BW Modes 20/40 Gbps 40/140 Gbps
Queues/Groups/Ports 8k/2k/768 64k/16k/128
Supported Fabric Cards 40G and 140G fabric 140G fabric only
PLIMs
8 x 10GE
16xOC48, 4xOC192, 1xOC768
Modular 6 x SPA
14 x 10GE
20 x 10GE (oversubscribed)
1 x 100GE
SONET and Modular PLIMs (future)
Power
LC (MSC + PLIM): 530W
Switch Card (4/8/16 slot): 102/185/206W
Total (4/8/16-slot): 3.5/6.6/11.5KW
LC (MSC + PLIM): 600W
Switch Card (4/18/16 slot): 49/90/94W
Total (4/8/16 slot): 4/7.5/14 KW
14
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Line Card Chassis (LCC) Fabric Card Chassis (FCC)
CRS-3 Full Rack MultiChassis
100m FRONT: 24 Fabric cards
2 Shelf Controllers
Control Ethernet Connections
BACK: 24 Optical Interconnect Modules (OIM)
2 OIM LED Module
Array Cable Connections
•Optical Backplane
•Redundant Fans/Power
•Supports CRS-16
FRONT: 16 Interface Slots
2 RP Slots
2 Controller slots BACK:
16 LC Slots
8 Fabric Card Slots
•Mid-Plane Design
•Redundant Fans/Power
•140G per Slot
• Each Line Card Shelf add 4.48 Tbps to the MultiChassis system
• Fabric architecture can support 72 LC Chassis and 8 Fabric Chassis - 322 Tbps
• Hitless Single Chassis to MultiChassis Upgrade 15
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Overview of CRS Data Path
High Level view Switch Fabric and Fabric attributes
Basic Fabric Building Blocks
CRS-1 and CRS-3 Fabric Bandwidth Comparison
High Level Data Path Ingress to Egress via Fabric
16
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Switch Fabric:
• Three Stage (S1, S2, S3) Benes Topology
• Multicast replication in Fabric
• 2 levels of priority through Fabric
HP Low latency path LP Best effort traffic
1 of 8
2 of 8
8 of 8
1
2
8
1
2
16 Line Card
MSC Line Card
MSC
S1 S3
S1 S3
S1 S3 S2
S2
S2
Cisco CRS Switch Fabric High Level
Ingress Side: Each Line Card (and RP) is connected to all Fabric planes through the S1 stage
Line cards chop Packets up into “Cisco Cells” for transport across the fabric
Destination line card (or RP) is prepended to the Cell
Egress Side: Each Line Card (and RP) is connected to all Fabric planes through S3 stage
Line cards reassemble Packets from “Cisco Cells” for egress packet processing
17
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Cisco CRS Switch Fabric Attributes
Redundancy
• Fabric implemented across 8 fabric planes
• Loss of any one fabric plane does not reduce capacity
• Loss of additional planes reduces capacity by 1/7 per plane
Hardware Capacity:
• Fabric implements at 1296 x 1296 buffered non-blocking switch
• Provides Capacity for 72 LCC
72 LCC * 16 Line cards = 1152
72 LCC * 2 Route Processors = 144
Multicast
• Full multicast capability in the fabric
• 1M fabric mcast groups
• Mcast cells are dropped if fabric is congested
• Mcast cells queued separately from unicast
1 of 8
2 of 8
8 of 8
1
2
8
1 2
16 Line Card
MSC
Line Card
MSC
S1 S3
S1 S3
S1 S3 S2
S2
S2
Queuing:
• Discrete Queues for control and data plane
• Integrated Congestion & flow control
18
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-3 Basic Fabric Building Blocks
160G
100G
100G
141G
113G
113G
IngressQ
FabricQ
FabricQ
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3
S3
1 of 8 2 of 8
3 of 8 4 of 8
5 of 8 6 of 8
7 of 8 8 of 8
IngressQ segments packets into cells
FabricQ reassembles packets from cells
Fabric routes cells to egress card
S2
19
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS Fabric Bandwidth Comparison
IngressQ
FabricQ
S1
S3
2.5G x 4 = 10Gbps ingress (per plane) 10G * 8 planes = 80Gbps total ingress
FabricQ
S3
2.5G
2.5G
2.5G
2.5G
2.5G
2.5G
2.5G
2.5G
2.5G
2.5G
2.5G
2.5G
IngressQ
FabricQ
S1
S3
FabricQ
S3
5G
5G
5G
5G
5G
2.5G * 8 = 20Gbps egress (per plane) 20G * 8 planes = 160Gbps total egress
5G * 8 = 40Gbps egress (per plane) 40G * 8 planes = 320Gbps total egress
5G
5G
5G
5G
5G
5G
5G
5G
MSC40 MSC140 Fabric Fabric
CRS-1 CRS-3 5G x 5 = 25Gbps ingress (per plane) 25G * 8 planes = 200Gbps total ingress
20
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Fabric Bandwidth Calculations
CRS-3 • IngressQ to S1
‒ = 200Gbps (raw bw) * 8b/10b (encoding) * 120/136 (cell overhead) = 141Gbps
• S3 to FabricQ
‒ = 320Gbps (raw bw) * 8b/10b (encoding) * 120/136 (cell overhead) = 225Gbps
or 113Gbps per FabricQ
CRS-1 • IngressQ to S1
‒ = 80Gbps(raw bw) * 8b/10b (encoding) * 120/136 (cell overhead) = 56Gbps
• S3 to FabricQ
‒ = 160Gbps(raw bw) * 8b/10b (encoding) * 120/136 (cell overhead) = 112Gbps or
56 Gbps per FabricQ
21
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS Fabric Bandwidths and Links Between
MSC40 and MSC140 MSC40 MSC140
To Fabric
Ingress
IngressQ has 32 links @2.5G for 40G
= 32 links * 2.5Gbps *8/10 (coding) *120/136 (cell tax) = 56Gbps
(This is for 8 planes)
Scaling IngressQ to 140G requires 40 links @5G
IngressQ to Fabric ASIC S1
= 40 links * 5Gbps *8/10 (coding) *120/136 (cell tax) = 141Gbps
(This is for 8 planes)
From Fabric
Egress
2 FabricQ ASICs per MSC, each with 2 RX Links per S3 ASIC. 64 links @ 2.5Gb
= 64 links * 2.5Gbps *8/10 (coding) *120/136(cell tax) = 100Gb
(this is for 8 planes)
100 Gb is split across 2 FabricQ ASICs on the LC. Egress forwarding capacity
is only 40Gb, so the LC must backpressure when overloaded
Fabric ASIC S3 to FabricQ
= 64 links * 5Gbps * 8/10 (coding) *120/136 (cell tax) = 225Gbps or 113Gbps per FabricQ.
(for 8 planes)
22
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
IngressQ
To-Fabric interface
Every MSC and RP transmits data to the fabric via an IngressQ ASIC
3072 fabric destinations with 2 priorities, plus 2 multicast queues
An IngressQ ASIC has 48 TX links @5Gbps
‒ 40 used for 200Gbps raw bandwidth
‒ 141 Gbps net bandwidth
TX links physically connect to each of the fabric planes
MAC 100G PHY
PLA
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3
S3
PSE
EgressQ
PSE
Intel CPU Sub-system
FabricQ
FabricQ
IngressQ 1 of 8 2 of 8
3 of 8 4 of 8
5 of 8 6 of 8
7 of 8 8 of 8
S2
23
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
S1 Plane 1
Ingr
ess
LC
•With 8 fabric planes and 6 connections to the S1 ASICs, each CRS-3 LC has 48 connections. 40 of these links are
used for carrying data traffic.
•40 connections @ 5Gb = 200Gb
•8b/10b encoding reduces this to 160Gb effective BW
•Cell tax means that the final result is ~140Gb
S1 Plane 3
S1 Plane 5
S1 Plane 7
S1 Plane 2
S1 Plane 4
S1 Plane 6
S1 Plane 8
IngressQ to S1
24
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
S1/S2/S3 ASIC‟s
Fabric switching ASIC
5Gbps SerDes with 2.5Gbps backwards compatibility mode
HP/LP unicast and HP/LP multicast queues
MAC 100G PHY
PLA
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
PSE
EgressQ
PSE
Intel CPU Sub-system
FabricQ
FabricQ
IngressQ 1 of 8 2 of 8
3 of 8 4 of 8
5 of 8 6 of 8
7 of 8 8 of 8
25
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
S1-> S2 -> S3 switching
Ingress linecard selects which FabricQ ASIC on the egress linecard a packet should be sent to.
The selection is encoded in the Cell Header when the packet is converted into Cisco Cells
S1 switches cells streams, load-balancing across the three S2 ASICs
S2 queues and routes the cell to the correct S3 based on the Cell header
S3 queues and routes the cell to the correct FabricQ based on the Cell header
72 links from S1 stage to the 3 S2 ASICs (24 links to each ASIC)
144 links from S2 stage to the 2 S3 ASICs (24 links to each ASIC)
S1
S1 S3
S3
S2
S2
S2
26
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Switch Fabric Multicast Replication
CRS provides efficient multicast replication via 3 operations
S2 can replicate cells to registered (via FGID) S3 SEAs
S3 can replicate cells to registered (via FGID) FabricQs
Egress SPP can replicate packets for each output port
1 million Fabric Group IDs
Program fabric for mcast
S2 and S3 lookup FGID
S2 S3 S1
S1 switches to all S2s
Mcast replication at S2 based on FGID
Replication at S3 based on FGID
27
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Egress CRS-3
Fabric Plane
•On egress, in the CRS-3 LCC chassis,
there are 2 S3 ASICs per fabric plane.
(the same S13 ASIC acts as S1 or S3
for different Fabric stages
•In the 16-slot chassis, each S3
receives 72 out of 144 TX links in
service.
•Each MSC receives 8 TX links per S3
„pair‟
•In the 16-slot chassis, each RP
receives 4 TX links per S3 pair
S1
S1 S3
S3 S2
S2
S2
28
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
FabricQ CRS-3
From-Fabric interface
Every MSC and RP receives data from the fabric via one or more FabricQ
ASICs
‒ MSCs have two FabricQ ASICs
‒ RPs have one FabricQ ASIC
A FabricQ ASIC has 32 RX links @5Gbps
‒ 160 Gbps raw, 113 Gbps net bandwidth
‒ MSC140/FP140 uses 2 FabricQs for 225 Gbps total
RX links physically connect to each of the fabric planes
MAC 100G PHY
PLA (Beluga)
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3 S3
S3 S3
S1
S1
S2
S2
S3
S3
PSE (Pogo)
EgressQ (Tor)
PSE (Pogo)
Intel CPU Sub-system
FabricQ (Crab)
FabricQ (Crab)
IngressQ (Seal)
1 of 8 2 of 8
3 of 8 4 of 8
5 of 8 6 of 8
7 of 8 8 of 8
S2
29
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
S3
S3 Plane 1
Egre
ss L
C
S3
S3 Plane 2
S3
S3 Plane 3
S3
S3 Plane 4
S3
S3 Plane 5
S3
S3 Plane 6
S3
S3 Plane 7
S3
S3 Plane 8
•In the 16-slot chassis, there are 2*S3 ASICs per plane connected to each MSC/RP
•2 FabricQ ASICs per MSC, each with 4 RX Links per S3 ASIC
•8 connections per linecard per fabric plane
•64 connections in total @ 5Gb = 320Gb
•8B/10B encoding reduces BW to 256Gb
•cell tax reduces this down to 225Gb
•225Gb is split across 2 FabricQ ASICs on the LC
S3 to FabricQ CRS-3
30
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Overview of CRS Switch Fabric
Fabric stages defined
Fabric Planes
Fabric Resilience
31
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Cisco CRS-1 Switch Fabric Overview High Speed Path for Transit Packets and IPC
1 of 8
2 of 8
8 of 8
1
2
8
40 Gbps 100 Gbps
Line Card Line Card
8 independent fabric planes 2.5x speedup through fabric Support for 72 chassis & 1296 RP/MSC clients
32
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Cisco CRS-1 Switch Fabric Overview
3 Stages with Priority
1 of 8
2 of 8
8 of 8
1
2
8
1
2
16
Line Card Line Card
S1 S2 S3
S1 S2 S3
S1 S2 S3
3 stage switching fabric
High and low priority cells Set on control by default Set on transit pkts via CLI Vital Bit for IPC
33
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-3 Switch Fabric - Simplified Form
The 8 fabric planes providing ingress BW is 200Gb which reduces to ~140Gb effective capacity due 8/10B encoding and cell tax overhead
On egress, each linecard has 2 ports per plane at 20Gb. 40Gb per fabric plane * 8 planes = 320Gb. 8B/10B encoding reduces BW to 256Gb. Cell tax reduces down to 225Gb
Egress PSE ASIC forwarding capacity is ~145Gb, so the LC must backpressure when overloaded
34
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Decisions at Each Stage in 16 slot CRS-3 For each of the 8 planes
S2 S3 S1
Send to any S2. No need to look at header
Look at cell header. Send to S3 based on chassis
Look at cell header. Send to specific LC and FabricQ
35
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-3 Fabric Planes
A full fabric consists of 8 planes
Switch Element ASICs (SEAs) perform switching operations
Each plane is an independent set of SEAs
A Switch Element ASIC can act as S1, S2 or S3 stages of Fabric. The S1 and S3 stages are in the same ASIC.
SEAs have 72 RX and 144 TX links
Number of RX and TX active depends on programming/stage
Cells never move between planes
Each plane will have multiple S13 (S1 and S3 stages) and S2 asics. E.g. It will have 2 S13 asics and 3 S2s. The S13 asic will act as both S1 and S3 stage on the LCC.
36
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS Fabric Resilience and System Recovery
CRS can operate with missing/failed fabric components
System can disable individual ―link‖ but still use plane
BW Capacity reduced if links or entire planes are down
Full capacity for Intermix traffic can be reached with 7 planes
At least 2 planes must be active for the system to work
Specifically one odd and one even numbered plane must be up.
Each cell is sent on a single plane
Cells have error correcting ECC, but
No multi-plane reconstruction of errored cells
Removing fabric HW w/o shutting down the plane first will lose cells
37
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MultiChassis Specifics
SC vs. MC (Single Chassis vs. MultiChassis)
Fabric details
MultiChassis benefits
MC building blocks
Fabric Plane config, cabling, and SFC placement
Sample MC Systems
CRS System Failure and Recovery
38
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS Single Chassis vs. MultiChassis
In MultiChassis, FCC provides the Fabric switch connectivity between
different Line Card Chassis which house the RPs, MSCs, PLIMs and
DRPs.
The Control Ethernet network is used for processes on different devices to
communicate for functions such as system device discovery, image
transfers, heartbeat messages, alarms and configuration management
Spanning Tree Protocol is used between the Switch elements on the Shelf
controller and on the inter-RP links in order to prevent loops
39
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MultiChassis and Benes Fabric
The distinct stages of the Benes fabric allow easy migration to MultiChassis
‒Line cards (MSC) and S1 and S3 stages stay on LCC
‒S2 Stage move to the FCC
‒Array Cables connect the fabric stages together
Upgrading from single chassis to MultiChassis can be done in-service
‒The S2 stage for each plane is moved to FCC one at a time
‒LCC requires new fabric cards with just the S1 and S3 stages
MSC
S1 S3
S1 S3
S2
S2
MSC
Up to 100m
Up to 100m
Line Card Chassis (LCC)
Line Card Chassis (LCC)
Fabric Card Chassis (FCC)
40
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS Fabric Chassis (FCC)
Front & rear access – mini-midplane for control network access
Front
‒24 Fabric card
‒2 Shelf Controllers
Back
‒24 OIM slots
‒2 Fiber LED modules
Dimensions:
‒23.6‖ W x 41*‖ D x 84‖ H
‒(60 W x 104.2 D x 213.36H (cm))
Power: ~9 KW DC, 11.1 KW AC
Weight: ~1550 lbs/704kg
Heat Dis.: 27600 BTUs
41
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
What benefits to expect from a CRS
MultiChassis System Scale
Process Placement to distribute the process load
Redundancy and Reliability
Performance
42
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
MultiChassis technology building blocks
Key points:
‒A fabric plane can span multiple S2 fabric boards
‒Multiple fabric planes can be supported in a Fabric chassis
1 0 2 3
4 5 6 7
1 0 2 3 5 4 6 7 0 1 2 3
4 5 6 7
1 FCC
2 FCCs
4 FCCs
43
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
MultiChassis technology - building blocks
0
8 FCCs
1 2 3
4 5 6 7
44
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Fabric Plane Configuration
Multi-Module Topology
Three Plane Configuration
Three S2 SFC‘s are used to create a plane
‒The array cables for each fabric plane are connected to all three cards
Each S13 card is connected to 3 different S2 cards
To LC rack # 0
To LC rack # 2
To LC rack # 8
To LC rack # 6
Single Module Topology
Full Plane Configuration
Each S2 SFC serves one plane The array cables for each fabric plane are connected to a
common SFC
Each S13 card is connected to a single S2 card
To LC rack # 1
To LC rack # 0
Switch Fabric Card (SFC) Switch Fabric Cards
45
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
MultiChassis Cabling
4+4 Configuration
‒4 CRS-16 Line Card Chassis
‒4 Fabric Card Chassis
2+1 Configuration 2 CRS-16 Line Card Chassis
1 Fabric Card Chassis
46
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
SFC – Switch Fabric Card Placement The CRS MutiChassis system supports the placement of all the OIMs and fabric
cards in a single chassis or across multiple chassis
‒CRS Fabric Chassis installation on cisco.com provides guidelines and best practices for fabric card placement
The CRS MultiChassis systems support configuration with one, two, or four Fabric Chassis
Additional chassis‘ add fabric redundancy, not additional capacity
‒Protects fabric bandwidth in case of the loss of single Fabric chassis
‒The CRS fabric only needs 7 planes to run at full capacity
‒The 8th plane provides active load sharing redundancy
‒The CRS fabric capacity degrades with losses of additional planes
‒ Loss of additional planes reduce throughput by approximately 1/7
The CRS fabric requires at least 1 odd plane and one even plane to function
MultiChassis capacity is gated by the OIM hardware
‒Up to 3 CRS-16 LCCs, in single topology mode where 8 OIM/SFC are required
‒Up to 9 CRS-16 LCCs, in multi-module topology mode where 24 OIM/SFC are required
47
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
OIM LED Module OIM LED Module provided to aid Operations
‒Two OIM LED Modules per Fabric Chassis
‒LEDs provide visual indication of the status of each array cable
‒Each tri-color LED shows 5 states:
OK (green)
no signal (none)
misconnected (blinking red)
signal fault (yellow)
connect here (slow blinking green)
OIM LED Module provides an installation aid to help identify misconnected fiber bundles
‒Based on fabric configuration, a point-point connection map will be generated by the router
‒If only 1 fiber bundle is misconnected, specifies the correct place to connect or reconnect a fabric cable.
OIM LED Module
48
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Fabric Placement with Two CRS-16 LCC
Single Topology Example
SFC 0
SFC 1
SFC 2
SFC 3
SFC 4
SFC 5
SFC 6
SFC 7
SFC 8
SFC 9
SFC 1
0
SFC 1
1
SFC 1
2
SFC 1
3
SFC 1
4
SFC 1
5
SFC 1
6
SFC 1
7
SFC 1
8
SFC 1
9
SFC 2
0
SFC 2
1
SFC 2
2
SFC 2
3
She
lf Co
ntro
ller 0
Sh
elf C
on
trolle
r 1
Full Rack FCC Front of Chassis
Full Rack FCC Back of Chassis
OIM
LED
1
OIM
LED
0
2+2 FCC Single Topology
SFC 0
SFC 1
SFC 2
SFC 3
SFC 4
SFC 5
SFC 6
SFC 7
SFC 8
SFC 9
SFC 1
0
SFC 1
1
SFC 1
2
SFC 1
3
SFC 1
4
SFC 1
5
SFC 1
6
SFC 1
7
SFC 1
8
SFC 1
9
SFC 2
0
SFC 2
1
SFC 2
2
SFC 2
3
She
lf Co
ntro
ller 0
Sh
elf C
on
trolle
r 1
Full Rack FCC Front of Chassis
Full Rack FCC Back of Chassis
OIM
LED
1
OIM
LED
0
Legend :
SFC 0
SFC 1
SFC 2
SFC 3
SFC 4
SFC 5
SFC 6
SFC 7
SFC 8
SFC 9
SFC 1
0
SFC 1
1
SFC 1
2
SFC 1
3
SFC 1
4
SFC 1
5
SFC 1
6
SFC 1
7
SFC 1
8
SFC 1
9
SFC 2
0
SFC 2
1
SFC 2
2
SFC 2
3
She
lf Co
ntro
ller 0
Sh
elf C
on
trolle
r 1
Full Rack FCC Front of Chassis
Full Rack FCC Back of Chassis
OIM
LED
1
OIM
LED
0
2+1 Single Topology
49
Plane 0
Plane 1
Plane 4
Plane 2
Plane 5
Plane 3
Plane 6
Plane 7
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Fabric Placement with Three CRS-16 LCC
MultiModule Topology Example
SFC 0
SFC 1
SFC 2
SFC 3
SFC 4
SFC 5
SFC 6
SFC 7
SFC 8
SFC 9
SFC 1
0
SFC 1
1
SFC 1
2
SFC 1
3
SFC 1
4
SFC 1
5
SFC 1
6
SFC 1
7
SFC 1
8
SFC 1
9
SFC 2
0
SFC 2
1
SFC 2
2
SFC 2
3
She
lf Co
ntro
ller 0
Sh
elf C
on
trolle
r 1
Full Rack FCC Front of Chassis
Full Rack FCC Back of Chassis
OIM
LED
1
OIM
LED
0
Dual FCC Configuration
SFC 0
SFC 1
SFC 2
SFC 3
SFC 4
SFC 5
SFC 6
SFC 7
SFC 8
SFC 9
SFC 1
0
SFC 1
1
SFC 1
2
SFC 1
3
SFC 1
4
SFC 1
5
SFC 1
6
SFC 1
7
SFC 1
8
SFC 1
9
SFC 2
0
SFC 2
1
SFC 2
2
SFC 2
3
She
lf Co
ntro
ller 0
Sh
elf C
on
trolle
r 1
Full Rack FCC Front of Chassis
Full Rack FCC Back of Chassis
OIM
LED
1
OIM
LED
0
SFC 0
SFC 1
SFC 2
SFC 3
SFC 4
SFC 5
SFC 6
SFC 7
SFC 8
SFC 9
SFC 1
0
SFC 1
1
SFC 1
2
SFC 1
3
SFC 1
4
SFC 1
5
SFC 1
6
SFC 1
7
SFC 1
8
SFC 1
9
SFC 2
0
SFC 2
1
SFC 2
2
SFC 2
3
She
lf Co
ntro
ller 0
Sh
elf C
on
trolle
r 1
Full Rack FCC Front of Chassis
Full Rack FCC Back of Chassis
OIM
LED
1
OIM
LED
0
Single FCC Configuration
Legend :
50
Plane 0
Plane 1
Plane 2
Plane 3 Plane 7
Plane 6
Plane 5
Plane 4
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-3 LC/Fabric(MultiChassis) connectivity
S2 Links S1 Links
S2 Fabric Card S13 Fabric Card S13 Fabric Card
LC
LC1
Seal Links IngressQ
S3
FabricQ
FabricQ
FabricQ
FabricQ
LC1
S1
S1
S3 S3 Links
S2
S2
S2
51
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Failure detection and System Recovery
The CRS system works towards detecting different type of failures in the system and take corrective action. Some of these failures could be:
Fabric Failure
Board Failure
Cable Failure
ASIC failure for Fabric, RP or Line card.
ASIC errors like ECC, Pairity or other miscellaneous errors.
Any single node non fabric failure e.g. RP, DRP, MSC.
Link errors and recovery
Rack failure for LCC
Rack failure for FCC
52
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Failure detection and System Recovery Contd...
Fabric failure
‒ The fabric board failures or cable failures are detected and reported by the fabric software and corrective action will be taken based on the severity of the failure in terms of
Plane MCAST_DOWN
All unicast traffic should pass through the plane except for destinations in the rack where the failure is based. This gives an oppurtunity for the system admins to recover from the malfunctioning board and cable and then bring the plane back into full action.
Plane DOWN
If the failure of fabric board or cables impacts the whole plane (e.g. An S2 board) , the plane is brought down completely so that traffic can be redirected through other planes till this failure is corrected and plane is functional again.
Note: Extensive FMEA testing has been done on all CRS-3 boards to detect, report and take corrective action of any possible board failure.
ASIC Failures and errors recovery
53
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MultiChassis Switch Fabric Details
MC Optical interconnects and cabling
MC SFC types (S13 and S2)
MC Fabric Topologies (Single module vs. Multi Module)
54
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS Switch Fabric Interconnect Optical Interconnect
FIBER BUNDLE
‒12 fibers per ribbon cable
‒6 ribbon cables per bundle = 72 fibers per Array cable
Multiple cables Between LCC Chassis and FCC Chassis (100m max)
Array cable connections on an S13 card
55
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS Array Cables Details
Array Cables connect FCC to the LCC
24 Array cables required for each LCC (3 per fabric plane x 8 fabric planes)
Array Cable composed of individual Fibers
Each cable consists of 6 ribbon cables with 12 fibers each
10m, 15m, 20m, 25m, 30m, 40m, 50m,60m,70m, 80m, 90m,100m length variants
Terminated at each end with a Square keyed Connector
56
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Array Cables
Turn Radius collar attachment
57
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS S13 Switch Fabric Card (SFC) For CRS-1 S13 contains the 2 S1 and 4 S3 ASICs from the S123 Card now
with parallel optical devices (PODs)
For CRS-3 S13 contains 2 ASIC‘s (S1,S3 mode)
For CRS-1 top S1 and S3 elements service linecard shelf slots 0->7 Bottom S1
and S3 elements service slots 8->15 plus 2 RP slots
For CRS-3 all LC and RP are connected to both SEA S3 ASIC‘s, three
connections to each Asic from each LC.
Each POD terminates a Fiber ribbon consisting of 12 unidirectional links each
operating at 2.5Gbps for CRS-1 and 5Gbps for CRS-3
The POD perform electrical to optical or optical to electrical conversions for
data for transmission over multimode fiber (850 nm), for distances of up to 100
meters. 6 PODs are used for the 72 fibers from S1 ASICs/12 PODs are
used for the 144 fibers from S3 ASICs
58
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS S13 SFC Card
S13 Service Processor
S1
S1
3* Tx POD 3 Fiber Ribbons
3 Fiber Ribbons
S3
S3 6* Rx POD
6* Rx POD
6 Fiber Ribbons
3* Tx POD
6 Fiber Ribbons S3
S3
59
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS S13 - Fiber to bulkhead mapping
S1
S1
S3
S3
S3
S3
S3 Fibers
S1 Fibers
60
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Fiber mapping
61
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS S2 Switch Fabric Card (SFC)
S2 Service Processor
S2
S2
6* Rx POD
12* Tx POD
6 Fiber Ribbons
12 Fiber Ribbons
S2
S2
6* Rx POD
12* Tx POD
6 Fiber Ribbons
12 Fiber Ribbons
S2
S2
6* Rx POD
12* Tx POD
6 Fiber Ribbons
12 Fiber Ribbons
• S2 Card consists of PSU, Service Processor and 3*S2 sub-boards
• 6 S2 ASICs and 54 parallel optical devices (PODs) per card for CRS-1
• 3 S2 ASICs and 54 parallel optical devices (PODs) per card for CRS-3
• Each POD terminates 1 Fiber ribbon (containing 12 fibers)
• 6 PODs per sub-board are used for the 72 fibers from the S1 ASICs
• 12 PODs per sub-board are used for the 144 fibers to the S3 ASICs
Bo
ard 0
B
oard
1
Bo
ard 2
62
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Fabric Chassis S2 Optical Interface Module (OIM)
• Passive device providing fiber X-connect function
• OIM distributes the fibers within each bundle to the S2 ASICs
• 9 Array cables per ‘single-wide’ OIM provides connectivity for up to 3 LCC chassis’
Larger size chassis’ deployments require cabling layout to be switched from vertical to horizontal
Note: OIM must be installed before S2 card insertion
63
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS S13 S2 card interconnect for MC single
module (vertical) cabling
S2 Service Processor
S2
S2
6* Rx POD
12* Tx POD
6 Fiber Ribbons
12 Fiber Ribbons
S2
S2
6* Rx POD
12* Tx POD
6 Fiber Ribbons
12 Fiber Ribbons
S2
S2
6* Rx POD
12* Tx POD
6 Fiber Ribbons
12 Fiber Ribbons
Rack 0 S13 Card Rack 1 S13 Card Rack 2 S13 Card
64
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC Optical Interface Module
To LC rack # 0
• A ‘single-wide’ Optical Interface Module is mated to one S2 Switch Fabric Card
• This configuration can support up to 3 chassis’ for single module (vertical) cabling
To LC rack # 2
65
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
0 1 2 3 S13 Fabric Slots
0 1 2 3 4 5 6 7 8 LCC0 - Plane 0
11 10 9 8 7 6 5 4 3 2 1 0 S2 Fabric Slots
Single Module Mode
FCC Chassis
A0
A1
A2
LCC0
LCC1
LCC2
CRS MC Fabric Topology
LCC0 Chassis
rear view right-to-left rear view left-to-right
FCC - Plane 0
66
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
0 1 2 3 fabric Slots
LCC0 - Plane 0
LCC0 Chassis
A0
A1
A2
0 1 2 3 4 5 6 7 8
11 10 9 8 7 6 5 4 3 2 1 0 Slots
FCC Chassis
Multi Module Mode
LCC1 LCC0
LCC3 LCC2
LCC5 LCC4
LCC8
LCC6 LCC7
CRS MC Fabric Topology
rear view left-to-right
FCC - Plane 0 rear view right-to-left
67
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS Optical Interface Module – Fiber distribution
• SINGLE-WIDE OIM
• Individual fiber connections are ‘wired’ to all S2 ASICs
• 3 TX fiber ribbons (36 links in total) are used to connect from each of the S1 ASICs in the LCC to the S2 ASICs in the FCC
• 36 links go to the CRS-1 S2 board, with 6 links to each of the 6 S2 ASIC on the board
• 36 links go to the CRS-3 S2 board, with 12 links to each of the 3 S2 ASICs on the board
Rack0 Plane X S1->S2
68
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS Optical Interface Module – Fiber distribution
• SINGLE-WIDE OIM
• Each S2 ASIC in CRS-1 FCC has 72 TX fiber links
• Each S2 ASIC in CRS-3 FCC has 144 TX fiber links
• 6 links are used to connect to each of the S3 ASICs
• 144 S2->S3 links per line card shelf per plane
• This configuration can service 12 S3 ASICs per plane
Rack0 Plane X S2->S3
69
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS OIM (horizontal cabling)
To LC rack # 0
• For up to 9 chassis’, a single fabric plane spans 3 S2 SFC’s
To LC rack # 2
To LC rack # 8
To LC rack # 6
cabling arrangement as follows:
70
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC Fabric Chassis - Cabling CONNECTION MAP
‒Based on configuration, a p2p connection map will be generated by the router
LED INSTALLATION AID
‒Flag misconnected fiber bundles
‒If only 1 fiber bundle is misconnected, identify correct plug hole (with some amount of
persistence)
‒5 states, 1 tri-color LED :
• OK (green)
• no signal (none)
• misconnected one cable (blinking red)
• misconnected more than one cable (red)
• signal fault (yellow)
• connect here (slow blinking green)
coresponds to blinking red
71
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC 2+1 System (vertically) cabled
FCC LCC 0 LCC 1
8 S2 boards & OIMs
8 S13 boards
72
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MultiChassis Fabric – Data Path
IngressQ S1
S2
FabricQ S3
Linecard LC Midplane
S13 Card Bundle Fiber Module
S2 Card
1. Data segmented. Cells distributed over 8 planes
2. Load balance to the available S2s in plane
3. Switch cell to correct S3. Multicast is replicated here.
4. Switch cell to FabricQ. 5. Data reassembled into packets
LCC FCC fiber
73
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
• Each CRS-3 LC has 6* links to the S1 asic. So total 48 links (8 planes). Each TX Link has a capability of carrying 5G of raw traffic.
• Each CRS-1 LC has 4 connections to the S1 ASICs on the fabric plane (8 planes = 32 links). Each TX Link is capable of carrying 2.5G of raw traffic.
• Each CRS-1 RP has 2 connections to the Lower S1 ASICs on the fabric plane only (8 planes = 16 links). This is 2.5G as well.
Fabric Plane
Upper shelf
Lower shelf
Ingress LC
Ingress LC
RP
S1
S1 S3
S3
S2
S2
S2
Ingress CRS 16-slot LCC
74
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC Fabric – Terminology
Link: ‒Data link between two fabric stages. Each link has two link ports, a transmitter and a receiver.
Bundle: ‒Cable between the LC and Fabric shelves. Each end terminates at a bundle port. Also known as Array cable.
Link port types Ingressqtx, s1rx ,s1tx, s2rx, s2tx, s3rx, s3tx and fabricqrx.
Link port inter-connect Ingressqtxs1rx, s1txs2rx, s2txs3rx, s3txfabricqrx.
75
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Select from LCC0 – Plane 0 – A0
Remove Dust Caps
Connect one end into Bundle A0
CRS MC Fabric Topology
76
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Connect LCC0 – Plane 0 – Bundles A0, A1, A2
A1
A0
A2
CRS MC Fabric Topology
77
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
A1
A0
A2
Connect bundle other end on FCC0 – Plane0
CRS MC Fabric Topology
78
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MultiChassis Control Ethernet
MC control Ethernet functions
MC control Ethernet HW – Integrated Shelf Controller
MC control Ethernet connectivity
79
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
MS
C
PL
IM
PL
IM
PL
IM
PL
IM
PL
IM
PL
IM
PL
IM
Fan\F
AB
RIC
(rear)
Power Supplies
Air Intake
Air Exhaust (r)
PL
IM
PL
IM
PL
IM
PL
IM
PL
IM
RP
/FA
BR
IC(r
ear)
PL
IM
PL
IM
LC
PL
IM
MS
C
PL
IM
MS
C
PL
IM
PL
IM
PL
IM
PL
IM
PL
IM
PL
IM
PL
IM
Fan
\FA
BR
IC(r
ear)
Power Supplies
Air Intake
Air Exhaust (r)
PL
IM
PL
IM
PL
IM
PL
IM
PL
IM
RP
/FA
BR
IC(r
ear)
PL
IM
PL
IM
LC
PL
IM
MS
C
PL
IM
24 cables to each LC chassis
RP1
Air Exhaust (r)
Air Exhaust (r)
SC
-GE
-22
S
C-G
E-2
2
12 x S2
Power Supplies
12 x S2
RP0
Stacking link
LCC0 LCC1 FCC
Connected to OIM’s (rear) on FCC
Connected to S13 cards (rear) on LCC
2LCC +1FCC MC Component connectivity via SC
80
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS-1 MultiChassis Control Ethernet
All communication from the Line card RPs to integrated switch is over the Control Ethernet
‒The integrated switch is not connected to the fabric
The Control Ethernet is used for many purposes
‒System Boot
‒Node availability (Heart beat) checks
‒All communication from the LCC to the FCC.
The Control Ethernet is redundant and must be connected in a fully meshed configuration to all active and standby RPs and SCs
‒2+1 Systems requires 9 cables – 8 RP to SCGE and 1 SCGE to SCGE
‒2+2 System requires 15 cables – 8 RP to SCGE and 6 SCGE to SCGE
‒2+4 System requires 36 cables – 8 RP to SCGE and 28 SCGE to SCGE
The Control Ethernet uses Spanning Tree (STP) to determine which paths to use for communication
81
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Integrated Shelf Controller
Product Description:
–Fabric chassis houses 2 Shelf Controllers (SC) acting as Primary and Secondary
–Provides local management of fabric chassis components
Boot and initialization of the SFCs, the optical interface module LED (OIM-LED) card, alarms, power supplies, and fans
–Integrated GE Ethernet Switch Interface
Provides out of band inter-chassis control network
Full Mesh connectivity required for control Ethernet
–Two SC-22GE are provided for redundancy
Product ID:
CRS-FCC-SC-22GE
82
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
2+1 MC External GE Connections
SC-GE-22
SC-GE-22
SC0
SC1
RP0
RP1
RP0
RP1
SC0-RP GE Links
SC1-RP GE Links
SC-SC GE Links
Note there is still an FE link over the backplane on the FC between the 2 SC-
GE-22 cards
LCC0
LCC1
FCC0
83
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
2+2 MC External GE Connections
SC-GE-22
SC-GE-22
SC0
SC1
RP0
RP1
RP0
RP1
LCC0
LCC1
FCC0
SC-GE-22
SC-GE-22
SC0
SC1
FCC1 84
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MultiChassis Configuration
MC dSC
MC SW distribution
MC rack configuration
MC fabric plane topology configuration
85
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC - Introducing the dSC
dSC = designated System Controller
dSC is responsible for overall system control, configuration and operation
Image download & synchronization to all devices in system is controlled by dSC
By default, dSC is the Primary RP in the first rack (LCC) that boots. If RP fails, Secondary RP assumes the role
If the LCC housing the dSC where to fail, today, MC system will reboot and dSC will come active on one of other LCC‘s connected to the system
Eventually, dSC functionality will be able to move between racks in a graceful manner
No specific configuration is required to become dSC
86
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
IOS-XR SW version on MC
dSC determines the IOS XR version on all components of the system
– Adding a new LCC with different IOS XR version will install the version running on the dSC
– Upgrading from single to MC – the FCC will install the IOS XR version running on the dSC
– Inserting new MSCs – same as on single chassis
87
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC – configuration Rack numbers
RP/0/RP0/CPU0:CRS3#admin show run | i dsc
Building configuration...
dsc serial TBA10080000 rack 1
dsc serial TBA10090001 rack 0
dsc serial TBA10120000 rack f0
• Assign a rack number to the S/N of each chassis • Serial numbers can be obtained from ‘sh diag chassis’ - From rommon with “dumpplaneeeprom” - From rear of the FCC and front of the LCC • Rack 0 or 1 = dSC (also an LCC) • Rack 1->127 = LCC • Rackf0-> f4 = FCC • Note: Rack numbers have to be unique Example:
88
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS LCC / FCC Serial Number Location
Watchout ! It is on the rear Side of the Fabric Chassis, unlike line card chassis
Front side of the Linecard Chassis
LCC FCC
89
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC Fabric – Config
Plane Topology configuration
Planes are not tied to specific slots in fabric rack
Admin-level config defines the slots each plane uses (and how big the plane is) controllers fabric plane 0
oim count 1
oim width 1
oim instance 0 location F0/SM0/FM
Count 1- All cables in plane connect to the same OIM. Count 3 - The cables from each LCC for that plane connect to different OIMs
Position of 1st card within the plane in fabric rack: plane 0 uses rack f0 slot sm0
90
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC Fabric Configuration Example RP/0/RP0/CPU0:CRS3#admin show run
Building configuration...
dsc serial TBA10080000 rack 1
dsc serial TBA10090001 rack 0
dsc serial TBA10120000 rack F0
controllers fabric plane 0
oim count 1
oim width 1
oim instance 0 location F0/SM0/FM
!
controllers fabric plane 1
oim count 1
oim width 1
oim instance 0 location F0/SM3/FM
!
controllers fabric plane 2
oim count 1
oim width 1
oim instance 0 location F0/SM6/FM
!
[SNIP]
controllers fabric plane 2
oim count 1
oim width 1
oim instance 0 location F0/SM21/FM
!
9 6 3 0
21 18 15 12
91
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Troubleshooting CRS MC
MC Control Ethernet connectivity verification & statistics
MC Control Ethernet UDLD and spanning tree functions
MC Shelf Controller (SC) LED‟s
Monitoring MC fabric plane, bundles and links
92
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC RP Connectivity
To verify the control ethernet connectivity on the RPs use: RP/0/RP0/CPU0:router(admin)#show controllers switch 0 ports location 0/RP0/CPU0
Ports Active on Switch 0
FE Port 0 : Up, STP State : FORWARDING (Connects to - 0/RP0)
FE Port 1 : Up, STP State : BLOCKING (Connects to - 0/RP1)
FE Port 2 : Up, STP State : FORWARDING (Connects to - 0/FC0)
FE Port 3 : Up, STP State : FORWARDING (Connects to - 0/FC1)
FE Port 4 : Up, STP State : FORWARDING (Connects to - 0/AM0)
FE Port 5 : Up, STP State : FORWARDING (Connects to - 0/AM1)
FE Port 6 : Down (Connects to - )
FE Port 7 : Down (Connects to - )
FE Port 8 : Up, STP State : FORWARDING (Connects to - 0/SM0)
FE Port 9 : Up, STP State : FORWARDING (Connects to - 0/SM1)
FE Port 10 : Up, STP State : FORWARDING (Connects to - 0/SM2)
FE Port 11 : Up, STP State : FORWARDING (Connects to - 0/SM3)
FE Port 12 : Up, STP State : FORWARDING (Connects to - 0/SM4)
FE Port 13 : Up, STP State : FORWARDING (Connects to - 0/SM5)
FE Port 14 : Up, STP State : FORWARDING (Connects to - 0/SM6)
FE Port 15 : Up, STP State : FORWARDING (Connects to - 0/SM7)
GE Port 0 : Up, STP State : FORWARDING (Connects to - GE_0)
GE Port 1 : Up, STP State : FORWARDING (Connects to - Switch 1)
93
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC SC Intra-rack Connectivity To verify the control ethernet connectivity on intra-rack switches on
the SC-GE-22s use: RP/0/RP0/CPU0:CRS-D(admin)#sh controllers switch 0 ports location F0/SC0/CPU0
FE Port 0 : Up, STP State : FORWARDING (Connects to - F0/SC0)
FE Port 1 : Up, STP State : BLOCKING (Connects to - F0/SC1)
FE Port 2 : Down (Connects to - F0/FC0)
FE Port 3 : Down (Connects to - F0/FC1)
FE Port 4 : Down (Connects to - F0/AM0)
FE Port 5 : Up, STP State : FORWARDING (Connects to - F0/AM1)
FE Port 6 : Up, STP State : FORWARDING (Connects to - F0/LM0)
FE Port 7 : Up, STP State : FORWARDING (Connects to - F0/LM1)
FE Port 8 : Down (Connects to - F0/SM0)
FE Port 9 : Up, STP State : FORWARDING (Connects to - F0/SM1)
FE Port 10 : Down (Connects to - F0/SM2)
FE Port 11 : Down (Connects to - F0/SM3)
FE Port 12 : Up, STP State : FORWARDING (Connects to - F0/SM4)
FE Port 13 : Down (Connects to - F0/SM5)
FE Port 14 : Up, STP State : FORWARDING (Connects to - F0/SM6)
FE Port 15 : Down (Connects to - F0/SM7)
GE Port 0 : Up, STP State : FORWARDING (Connects to - GE_0)
GE Port 1 : Up, STP State : FORWARDING (Connects to - Switch 1)
94
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC RP UDLD
UDLD runs on links between RPs and SCs. Use the UDLD
CLI to find out who is connected – this is a nice extra
feature of UDLD
RP/0/RP0/CPU0:ios(admin)#show controllers switch udld location 0/rp0/CPU0 … Interface GE_Port_0
…
…
Current bidirectional state: Bidirectional
Current operational state: Advertisement - Single neighbor detected
…
Entry 1
---
Device name: nodeF0_SC0_CPU0
Port ID: Gig port# 13
Neighbor echo 1 device: 0_RP0_CPU0_Switch
Neighbor echo 1 port: GE_Port_0
…
95
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC SC Intra-rack Connectivity
Use the UDLD CLI the same way as for RPs
RP/0/RP0/CPU0:CRS-D(admin)#sh controllers switch udld location F0/SC0/CPU0
Interface GE_Port_0
---
Port enable administrative configuration setting: Enabled
Port enable operational state: Enabled
Current bidirectional state: Bidirectional
Current operational state: Advertisement - Single neighbor detected
Message interval: 7
Time out interval: 5
Entry 1
---
Expiration time: 15
Device ID: 1
Current neighbor state: Bidirectional
Device name: nodeF0_SC0_CPU0
Port ID: Gig port# 22
Neighbor echo 1 device: F0_SC0_CPU0_Switch
Neighbor echo 1 port: GE_Port_0
96
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC SC Inter-rack Connectivity To verify the control ethernet connectivity on inter-rack
switches on the SC-GE-22s use: RP/0/RP0/CPU0:CRS-D(admin)#sh controllers switch inter-rack ports all location F0/SC0/CPU0
GE_Port_0 : Up
GE_Port_1 : Down
GE_Port_2 : Up
GE_Port_3 : Up
[SNIP]
GE_Port_17 : Down
GE_Port_18 : Down
GE_Port_19 : Down
GE_Port_20 : Down
GE_Port_21 : Up
97
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC RP STP STP is run on links between RPs and SCs. Use the STP
CLI to find out spanning tree info
RP/0/RP0/CPU0:CRS-D(admin)#sh controllers switch stp location 0/RP0/CPU0
##### MST 0 vlans mapped: 2-4094
Bridge address 0011.9362.a26c priority 36864 (36864 sysid 0)
Root address 5246.48f0.20ff priority 32768 (32768 sysid 0)
port GE_Port_0 path cost 0
Regional Root address 5246.48f0.20ff priority 32768 (32768 sysid 0)
internal cost 20000 rem hops 3
Operational hello time 1, forward delay 6, max age 8, txholdcount 6
Configured hello time 1, forward delay 6, max age 8, max hops 4
Interface Sts Role Cost Prio.Nbr Type
---------------- ---- ---- --------- -------- ------------------------------
##### MST 1 vlans mapped: 1
Bridge address 0011.9362.a26c priority 36865 (36864 sysid 1)
Root address 5246.48f0.20ff priority 32769 (32768 sysid 1)
port GE_Port_0 cost 20000 rem hops 3
Interface Sts Role Cost Prio.Nbr Type
---------------- ---- ---- --------- -------- ------------------------------
FE_Port_1 FWD Desg 200000 128. 2 P2p
GE_Port_0 FWD Root 20000 128. 49 P2p
98
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC SC Intra-rack Connectivity
Use the STP CLI the same way as for RPs
RP/0/RP0/CPU0:CRS-D(admin)#sh controllers switch stp location F0/SC0/CPU0
##### MST 0 vlans mapped: 2-4094
Bridge address 0800.453e.47b4 priority 36864 (36864 sysid 0)
Root address 5246.48f0.20ff priority 32768 (32768 sysid 0)
port GE_Port_0 path cost 0
Regional Root address 5246.48f0.20ff priority 32768 (32768 sysid 0)
internal cost 20000 rem hops 3
Operational hello time 1, forward delay 6, max age 8, txholdcount 6
Configured hello time 1, forward delay 6, max age 8, max hops 4
Interface Sts Role Cost Prio.Nbr Type
--------------- ---- ---- --------- -------- ------------------------------
##### MST 1 vlans mapped: 1
Bridge address 0800.453e.47b4 priority 36865 (36864 sysid 1)
Root address 5246.48f0.20ff priority 32769 (32768 sysid 1)
port GE_Port_0 cost 20000 rem hops 3
Interface Sts Role Cost Prio.Nbr Type
---------------- ---- ---- --------- -------- ------------------------------
FE_Port_1 BLK Altn 200000 128. 2 P2p
GE_Port_0 FWD Root 20000 128. 49 P2p
99
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC SC Inter-rack Connectivity Use UDLD CLI to find out who is connected
RP/0/RP0/CPU0:ios(admin)#show controllers switch inter-rack udld all location f0/sc0/CPU0
Interface Gig port# 0
---
Port enable administrative configuration setting: Enabled
Port enable operational state: Enabled
Current bidirectional state: Bidirectional
Current operational state: Advertisement - Single neighbor detected
Message interval: 7
Time out interval: 5
Entry 1
---
Expiration time: 14
Device ID: 1
Current neighbor state: Bidirectional
Device name: 0_RP0_CPU0_Switch
Port ID: GE_Port_0
Neighbor echo 1 device: nodeF0_SC0_CPU0
Neighbor echo 1 port: Gig port# 0
Message interval: 7
Time out interval: 5
CDP Device name: BCM_SWITCH
Interface Gig port# 2
--- 100
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC SC Inter-rack Connectivity Use STP CLI to find out STP information
RP/0/RP0/CPU0:ios(admin)#show controllers switch inter-rack stp location f0/sc0/cpu0
##### MST 0 vlans mapped: 2-4094
Bridge address 5246.48f0.20ff priority 32768 (32768 sysid 0)
Root this switch for the CIST
Operational hello time 1, forward delay 6, max age 8, txholdcount 6
Configured hello time 1, forward delay 6, max age 8, max hops 4
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- ---------------------------
##### MST 1 vlans mapped: 1
Bridge address 5246.48f0.20ff priority 32769 (32768 sysid 1)
Root this switch for MST1
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- ---------------------------
GE_13 Desg FWD 20000 128. 14 P2p
GE_14 Desg FWD 20000 128. 15 P2p
GE_15 Desg FWD 20000 128. 16 P2p
GE_17 Desg FWD 20000 128. 18 P2p
GE_22 Desg FWD 20000 128. 23 P2p
101
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC Show tech control-ethernet
Use this CLI to collect control ethernet logs for TAC
RP/0/RP0/CPU0:IOX(admin)#show tech-support control-ethernet file ..
…
…
…
102
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC SC-GE-22 Counters Checking port statistics counters
RP/0/RP0/CPU0:ios(admin)#show controllers switch inter-rack statistics all brief
location f0/sc0/cpu0
Port Tx Frames Tx Errors Rx Frames Rx Errors
-----------------------------------------------------------
GE_Port_0 : 374423 0 1776848 0
GE_Port_1 : 251232 0 170742 0
GE_Port_2 : 857923 0 414409 0
GE_Port_3 : 239437 0 152772 0
GE_Port_4 : 166166 0 82031 0
GE_Port_5 : 0 0 0 0
<SNIP>
GE_Port_16 : 0 0 0 0
GE_Port_17 : 0 0 0 0
GE_Port_18 : 0 0 0 0
GE_Port_19 : 0 0 0 0
GE_Port_20 : 0 0 0 0
GE_Port_21 : 0 0 0 0
Intra-rack : 522072 0 293720 0
Stacking : 1482 0 0 0
Stacking : 0 0 0 0
103
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC SC-GE-22 Counters
Clear statistics using
Per port:
RP/0/RP0/CPU0:ios(admin)#clear controller switch inter-rack statistics ports 0
location f0/sc0/cpu0
Or all ports:
RP/0/RP0/CPU0:ios(admin)#clear controller switch inter-rack statistics all location
f0/sc0/cpu0
104
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC SC-GE-22 LEDs
The SC-GE-22 has LEDs on the front panel for every port. The LEDs can be
used to get information about the link such as
‒Green Link Up
‒Blinking Green Activity
‒Amber Port Error Disabled (by UDLD)
‒Off Link Down
‒What color is stp blocking state? – it still stays green. In blocking state you are still
receiving udld/stp packets. The amber condition only indicates a fault on the port
‒Note: Admin Shutdown of ports is not supported
105
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC SC UDLD error messages
Whenever a port is disabled in the control network due to it being detected
as Uni-directional, a Syslog message is generated
The state of the port is displayed in ―show controller switch .. udld ..loc <>‖
CLI
The fiber and SFP should be checked on the port
You can try to bring-up the port and clear the error condition using
‒―clear controller switch [inter-rack] errdisable port …‖
Note that the port will be err-disabled again if the error condition still exists
106
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC SC Spanning tree
The following is the default priority order of nodes to become the root of the network
‒F0/SC0
‒F0/SC1
‒F1/SC0
‒F1/SC1
‒F2/SC0
‒F2/SC1
‒...
‒we use bridgeIDs to compare – priorities are same and mac address determine. Higher number is lower priority for STP, lower number wins. Note we don‘t us a guma (globally unique mac add) so we generate it geographically and can set the above up. Note that ma comes from the location of the card, its not burnt into the board itself, its generated at runtime during bootup
The LCC RP‘s are given lower priority than all FCC SC‘s so they will not become the root of the network as long as a single SC is booted up
107
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC Spanning tree – to see the root Use the following command to find the STP status on a
node. Verify that the root is the node you expect it to be RP/0/RP0/CPU0:ios(admin)#show controllers switch stp location f0/sc1/CPU0
##### MST 0 vlans mapped: 2-4094
Bridge address 0800.453e.468f priority 36864 (36864 sysid 0)
Root address 5246.48f0.20ff priority 32768 (32768 sysid 0)
port GE_Port_0 path cost 0
Regional Root address 5246.48f0.20ff priority 32768 (32768 sysid 0)
internal cost 40000 rem hops 2
Operational hello time 1, forward delay 6, max age 8, txholdcount 6
Configured hello time 1, forward delay 6, max age 8, max hops 4
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- ---------------------
##### MST 1 vlans mapped: 1
Bridge address 0800.453e.468f priority 36865 (36864 sysid 1)
Root address 5246.48f0.20ff priority 32769 (32768 sysid 1)
port GE_Port_0 cost 40000 rem hops 2
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- ---------------------
FE_Port_0 Altn BLK 200000 128. 1 P2p
GE_Port_0 Root FWD 20000 128. 49 P2p
108
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
CRS MC Spanning tree
in a normal topology Verify that F0/SC0 inter-rack is the root of the network and all spanning
tree port states seem normal
Verify that other SCs in the system see F0/SC0 as the root of the network
and are designated on links connecting to RPs
Verify that the RP has selected one of the GE ports as its‘ root port and is
‗alternate‘ on the other GE port
Verify that one of the RPs is blocked on the FE link connecting the RPs
109
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Verifying MC control network with “ping”
―Ping control-eth‖ from admin mode can be used for troubleshooting control-eth issues
RP/0/RP0/CPU0:CRS-D(admin)#ping control-eth location f0/sc0/cpu0
Src node: 513 : 0/RP0/CPU0
Dest node: 983553 : F0/SC0/CPU0
Local node: 513 : 0/RP0/CPU0
Packet cnt: 1 Packet size: 128 Payload ptn type: default (0)
Hold-off (ms): 1 Time-out(s): 2 Max retries: 5
DelayTimeout: 1Destination node has MAC addr 5246.480f.0201
Running CE node ping.
Please wait...
Src: 513, Dest: 983553, Sent: 1, Rec'd: 1, Mismatched: 0
Min/Avg/Max RTT (usecs): 1000/1000/1000
CE node ping succeeded for node: 983553
110
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Monitoring CRS MC Fabric Operation Error counts
RP/0/RP0/CPU0:CRS(admin)#show controllers fabric plane all statistics
In Out CE UCE PE
Plane Cells Cells Cells Cells Cells
---------------------------------------------------------------------------
0 45088937799 45088965120 0 0 0
1 45088939239 45088966386 0 0 0
2 45088554913 45088582368 0 0 0
3 45088556270 45088583448 0 0 0
4 45088151158 45088178280 0 0 0
5 45088152592 45088179766 0 0 0
6 45089166942 45089194313 0 0 0
7 45089168302 45089194928 0 0 0
CE Cells – Correctable Error UCE – Uncorrectable Error PE – Parity Error
111
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Monitoring CRS MC Fabric Operation High level fabric status
RP/0/RP0/CPU0:CRS(admin)#show controllers fabric plane all detail
Plane Admin Oper Down Total Down Id State State Flags Bundles Bundles ------------------------------------------------------ 0 UP UP 0 0 1 UP UP 0 0 2 UP UP 0 0 3 UP UP 0 0 4 UP UP 0 0 5 UP UP 0 0 6 UP UP 0 0 7 UP UP 0 0
Examples of Down flags P – plane admin down p – plane oper down C – card admin down c – card oper down
112
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Check the CRS MC fabric bundles All fibers are UP on each bundle ?
RP/0/RP0/CPU0:ios(admin)#sh controllers fabric bundle all detail
Flags: P - plane admin down, p - plane oper down
C - card admin down, c - card oper down
L - link port admin down, l - linkport oper down
A - asic admin down, a - asic oper down
B - bundle port admin Down, b - bundle port oper down
I - bundle admin down, i - bundle oper down
N - node admin down, n - node down
o - other end of link down d - data down
f - failed component downstream
m - plane multicast down
Bundle Oper Down Plane Total Down Bundle Bundle
R/S/M/P State Flags Id Links Links Port1 Port2
----------------------------------------------------------------------
F0/SM0/FM/0 UP 3 72 0 F0/SM0/FM/0 0/SM3/SP/0
F0/SM0/FM/1 UP 3 72 0 F0/SM0/FM/1 0/SM3/SP/1
F0/SM0/FM/2 UP 3 72 0 F0/SM0/FM/2 0/SM3/SP/2
F0/SM0/FM/3 UP 3 72 0 F0/SM0/FM/3 1/SM3/SP/0
F0/SM0/FM/4 UP 3 72 0 F0/SM0/FM/4 1/SM3/SP/1
F0/SM0/FM/5 UP 3 72 0 F0/SM0/FM/5 1/SM3/SP/2
F0/SM0/FM/6 DOWN bo 3 72 72 F0/SM0/FM/6 2/SM3/SP/0
F0/SM0/FM/7 DOWN bo 3 72 72 F0/SM0/FM/7 2/SM3/SP/1
F0/SM0/FM/8 DOWN bo 3 72 72 F0/SM0/FM/8 2/SM3/SP/2
[SNIP]
113
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Check the CRS MC fabric DOWN links F0/SM6/FM/1 UP 1 72 1 F0/SM6/FM/1 0/SM1/SP/1
(admin)#sh controllers fabric link port s2tx all | i UP.*DOWN.*SM1/SP/1
F0/SM6/SP/1/61 UP DOWN do 1/SM1/SP/1/25 F0/SM6/3 1/SM1/0
(admin)#sh controllers fabric link port s3rx all | i 1/SM1/SP/1/25
1/SM1/SP/1/25 UP DOWN l F0/SM6/SP/1/61 1/SM1/0 F0/SM6/3
(admin)#sh controllers fabric link port s2tx F0/SM6/SP/1/61 detail
[SNIP]
Sfe Port Admin Oper Down Sfe BP Port BP Other
R/S/M/A/P State State Flags Role Role End
----------------------------------------------------------------
F0/SM6/SP/1/61 UP DOWN do 1/SM1/SP/1/25
Connection Details for s2tx/F0_SM6_SP,0x1,0x3d
---------------------------------------
Type: Inter-chassis bundle
Near-end bundle port: bport/F0/SM6/3 ribbon 3 fiber 1
Far-end bundle port : bport/1/SM1/0 ribbon 2 fiber 1
HBMT pin name : P6L2_1
Fabric group offset : 0
Fabric group : 2
(admin)#sh controllers fabric link port s3rx 1/SM1/SP/1/25 detail
Start to clean here
114
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Check the CRS MC Bundle Statistics RP/0/RP0/CPU0:ios(admin)#show controllers fabric bundle port all statistics
Total racks: 3
Rack 0:
Bundle Port In Out CE UCE PE
R/S/M/P Cells Cells Cells Cells Cells
--------------------------------------------------------------------------------
0/SM0/SP/0 10032966 2237486 1484566 6 0
0/SM0/SP/1 4571827 2237495 776173 0 0
Rack 1:
Bundle Port In Out CE UCE PE
R/S/M/P Cells Cells Cells Cells Cells
--------------------------------------------------------------------------------
1/SM0/SP/0 7198601 10115782 7015647 910 0
1/SM0/SP/1 6978240 7649778 92393711 13972 0
Rack F0:
Bundle Port In Out CE UCE PE
R/S/M/P Cells Cells Cells Cells Cells
--------------------------------------------------------------------------------
F0/SM4/FM/0 159802 236890 0 0 0
F0/SM4/FM/1 159766 236825 21 10 0
115
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
More useful commands - CRS MC fabric
Fabric
‒ (admin)#show controllers fabric rack all detail
‒ (admin)#show controllers fabric plane all detail
‒ (admin)#show controllers fabric connectivity all detail
‒ (admin)#show controllers fabric plane all statistics
116
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Cisco CRS MultiChassis is managed as a system.
117
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Recommended Reading
118
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Evaluation Forms
Please fill out evaluation forms and surveys
BRKARC-3002
119
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Complete Your Online
Session Evaluation Give us your feedback and you
could win fabulous prizes.
Winners announced daily.
Receive 20 Passport points for each
session evaluation you complete.
Complete your session evaluation
online now (open a browser through
our wireless network to access our
portal) or visit one of the Internet
stations throughout the Convention
Center.
Don‘t forget to activate your
Cisco Live Virtual account for access to
all session material, communities, and
on-demand and live activities throughout
the year. Activate your account at the
Cisco booth in the World of Solutions or visit
www.ciscolive.com.
120
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Final Thoughts
Get hands-on experience with the Walk-in Labs located in World of
Solutions, booth 1042
Come see demos of many key solutions and products in the main Cisco
booth 2924
Visit www.ciscoLive365.com after the event for updated PDFs, on-
demand session videos, networking, and more!
Follow Cisco Live! using social media:
‒ Facebook: https://www.facebook.com/ciscoliveus
‒ Twitter: https://twitter.com/#!/CiscoLive
‒ LinkedIn Group: http://linkd.in/CiscoLI
121
© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
SC to MC conversion can be done : online (most customers) & offline
MC Downgrade to a SC is not supported
Online Conversion :
SC is live & carrying production traffic
CRS Fabric can route line rate IMIX traffic even with 7 planes UP
SC to MC online migration uses the above concept to migrate each plane one by one (S123 boards to S13 boards)
Though done in maintenance window, the control plane traffic is still live
Offline Conversion :
Customers who do not want to change metrics config to divert the traffic
Easier process if the operational folks in the POP & N/W admins in the Central Site
Least risky
Appendix – Typical Migration Steps SC to MC
Types of Migration : Online vs Offline
123
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix A – Case Study 1:
Offline SC to MC migration - Network Topology SBCR (Super Block Core Router)
BCR (Block Core Router)
AER (Area Edge Router)
SSE
L2SW
CO
RE
EDG
E A
CC
ESS
Aggregation routers of East connecting to West
Aggregation routers of each region
Aggregation routers of each prefecture
Multi-Service customer edge router
L2 aggregation switch
MC Installation
Site A
Site B
124
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix A – Case Study 1
MC Migration Plan Take one week maintenance window for one site
Setup 1+1 MC config for system verification
No live migration (Off-line upgrade from single chassis to multi chassis)
SSE SSE
Site A Site B
SSE SSE
Site A (Primary) Site B (Secondary)
125
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix A – Case Study 1
MC Migration Steps
SSE SSE SSE SSE
Site A (Primary) Site B (Secondary) Site A Site B
SSE SSE
Site A Site B
SSE SSE
Site A Site B
1 2
3 4
Install new LCC and FCC Setup 1+1 MC configuration Verify 1+1 MC system
Offline
Single to Multi Migration
dSC FCC
dSC FCC ndSC
126
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix A – Case Study 1
MC Migration Steps – contd.
SSE SSE
Site A Site B
SSE SSE
Site A Site B
SSE SSE
Site A Site B
5 6
7 8
Install new LCC and FCC Setup 1+1 MC configuration Verify 1+1 MC system
Offline
Single to Multi Migration
SSE SSE
Site A Site B
Live dSC FCC
dSC FCC ndSC
127
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix A – Case Study 1
MC Migration Scenario – contd.
9 SSE SSE
Site A Site B
Live
128
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix A – Case Study 1
MC Installation Procedure – summary
LCC0 FCC0 LCC1
dSC
LCC0 FCC0 LCC1
dSC
MC 1+1
MC 2+1
Exisiting LCC
First build up a MC 1+1 (LCC Rack num = 1) and test it offline. Step1. LCC1 Setup Step2. LCC1 Turboboot Step3. FCC0 Bootup Step4. Connecting Array cables Step 5. Then connect the existing LCC(Rack Num = 0) into
MC 1+1 offline (more details in next slide)
Assumption: LCC0 is the Line Card Chassis that is on and in operation. LCC1 is the new Line Card Chassis. FCC0 is the new Fabric Chassis. LCC0 will NOT be touched when building a Rack 1 1+1 Multi-Chassis.
129
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix A – Case Study 1
MC Installation Procedure - detail
LCC0 FCC0 LCC1
MC 1+1 Exisiting LCC
Step5. Connecting 1+1 MC into SC offline -Configure SC with SN of LCC1 & FCC -Configure LCC1 to be in install-mode -Bring down 1+1 MC -Replace all S123 boards to S13 & connect LCC0 to FCC -Connect the CE ports of LCC0 & 1 on FCC -Turn ON entire MC & allow it to bake -Ensure the Fabric is baked & ready -Run basic tests & ensure HW is fine on LCC1 -Remove “install-mode” from LCC1 -MC migration complete
XR RUN 3.6.1
XR RUN 3.6.1
Migrate S123 to S13 & connect cables
Turn ON
Verify LCC1 operation
Configure SN of LCC1 & FCC, install-mode
Remove LCC1 from install-mode
XR RUN 3.6.1
Verify MC operation
Power OFF Power OFF
Connect CE port Connect CE port Connect CE port
Power OFF
Turn ON Turn ON
130
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix B – Case Study 2: Online SC to MC
migration: MC Installation Procedure – summary
LCC0 FCC0 LCC1
dSC
LCC0 FCC0 LCC1
dSC
MC 1+1
MC 2+1
Exisiting LCC
First build up a MC 1+1 (LCC Rack num = 1) and test it offline. Step1. LCC1 Setup Step2. LCC1 Turboboot Step3. FCC0 Bootup Step4. Array Cable Connecting Step 5. Then connect the existing LCC(Rack Num = 0) into
MC 1+1 online (more details in next slide)
Assumption: LCC0 is the Line Card Chassis that is on and in operation. LCC1 is the new Line Card Chassis & will be put in install-mode FCC0 is the new Fabric Chassis. LCC0 will NOT be touched when building a Rack 1 1+1 Multi-Chassis.
131
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix B – Case Study 2
MC Installation Procedure - detail
LCC0 FCC0 LCC1
MC 1+1 Exisiting LCC
Step5. Connecting 1+1 MC into SC online -Bring down 1+1 MC -Connect the CE ports of LCC0 & 1 on FCC -Configure SC with SN of LCC1 & FCC -Configure LCC1 to be in install-mode -Turn ON FCC & allow it to bake -Ensure the Fabric is baked & ready -Migrate S123 boards on SC to S13 plane by plane -Once the SC is converted into LCC0 with all S13’s -Turn ON the LCC1 & allow it to bake -Run basic tests & ensure HW is fine on LCC1 -Remove “install-mode” from LCC1 -MC migration complete
XR RUN 3.6.1
XR RUN 3.6.1
Migrate S123 to S13 plane by plane
Turn ON FCC --> Fabric ready
Verify LCC1 operation
Turn ON LCC1
Configure SN of LCC1 & FCC, install-mode
Remove LCC1 from install-mode
XR RUN 3.6.1
Verify MC operation
Power OFF Power OFF
Connect CE port Connect CE port Connect CE port
132
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix C – FCC Physical Install notes
Optical Interface Module (OIM) - Rear
Back View
*Important: Always insert
OIM before inserting
corresponding S2 Fabric
Board. When removing
OIM, disengage S2 Fabric
Board first.
OIM
133
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix C – Physical Install notes
Tips for handling S2 Fabric Board
S2 Fabric Board is the longest and heaviest board.
Take care when carrying the board and swinging in an arc, not to hit the optical connectors.
Recommended hand placement when carrying the board.
Insert after OIM module.
134
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix C – Physical Install notes
Watch out for Bend Radius in Cable Trays
135
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix C – Physical Install notes Optics
Troubleshooting/Cleaning
S13 Manual Optics Cleaning
optic adapter
136
© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public
Appendix C – Physical Install notes
Optic Troubleshooting/Cleaning
OIM Manual Optics Cleaning
137