Nexus Validation Test Phase 4 - Cisco · 4 Service Provider MPLS: The topology validates the inter...
Transcript of Nexus Validation Test Phase 4 - Cisco · 4 Service Provider MPLS: The topology validates the inter...
1
NexusValidationTestPhase4.0
2
1 Introduction.....................................................................................................................................................3
2 NVTTopologyandScaleAchieved...........................................................................................................52.1 VxLANSolution...................................................................................................................................................................52.2 MPLSProfile1withF3,M1andM2andFabricPathandCEAccess........................................................62.3 ServiceProviderMPLSF3..............................................................................................................................................82.4 DynamicFabricAutomation(Vinci)Solution...................................................................................................102.5 DataCenter1...................................................................................................................................................................122.6 Enterprise1.......................................................................................................................................................................132.7 MSDCScaleProfiles.......................................................................................................................................................142.8 M1vPC.................................................................................................................................................................................16
3 ISSUMatrix....................................................................................................................................................17
4 ProfileDetails...............................................................................................................................................184.1 VxLANSolution................................................................................................................................................................184.1.1 NetworkTopologyandDesignReview..............................................................................................184.1.2 HardwareandSoftwareOverview.......................................................................................................214.1.3 VxLANSolutionConfigurationGuide..................................................................................................214.1.4 VxLANSolutionScale.................................................................................................................................21
4.2 MPLSProfile1withF3,M1andM2andFabricPathandClassicalEthernetAccess.....................224.2.1 MPLSTopologyDesignReview..............................................................................................................224.2.2 HardwareandSoftwareOverview.......................................................................................................234.2.3 Topology..........................................................................................................................................................244.2.4 SPMPLSScaleCoverage...........................................................................................................................244.2.5 SPMPLSISSUMatrix..................................................................................................................................25
4.3 SPMPLSSolution............................................................................................................................................................254.3.1 NetworkLogicalTopologyDesignOverview...................................................................................254.3.2 Descriptionofthetestnetwork.............................................................................................................264.3.3 HardwareandSoftwareOverview.......................................................................................................264.3.4 Topology..........................................................................................................................................................274.3.5 Scale...................................................................................................................................................................274.3.6 ISSUMatrix.....................................................................................................................................................284.3.7 TrafficandConvergence...........................................................................................................................28
4.4 DynamicFabricAutomation(Vinci).....................................................................................................................294.4.1 TopologyDesignReview..........................................................................................................................294.4.2 HardwareandSoftwareOverview.......................................................................................................304.4.3 Topology..........................................................................................................................................................314.4.4 TrafficandMetrics......................................................................................................................................31
4.5 DC1........................................................................................................................................................................................324.5.1 HardwareandSoftwareOverview.......................................................................................................324.5.2 NetworkLogicalTopologyDesignOverview...................................................................................324.5.3 ConfigurationDetails.................................................................................................................................33
4.6 ENT.......................................................................................................................................................................................364.6.1 HardwareandSoftwareOverview.......................................................................................................364.6.2 NetworkLogicalTopologyDesignOverview...................................................................................36
4.7 ConfigurationDetails...................................................................................................................................................374.8 ENTTopology...................................................................................................................................................................384.9 MSDCScale........................................................................................................................................................................394.9.1 TopologyDesignOverview......................................................................................................................394.9.2 ISSUMatrix.....................................................................................................................................................40
3
4.10 M1vPCScale..................................................................................................................................................................404.10.1 HardwareandSoftwareOverview....................................................................................................404.10.2 NetworkLogicalTopologyDesignOverview................................................................................404.10.3 ConfigurationDetails...............................................................................................................................41
4.11 M1vPCScaleTopology.............................................................................................................................................424.12 DC1,ENT1andM1vPCFeature/ScaleCoverage.......................................................................................424.13 ISSUMatrixforDC1,ENTandM1vPC..............................................................................................................43
5 NVTFindings/Conclusion/Recommendations.................................................................................44
6 Appendix........................................................................................................................................................476.1 VxLANSolutionConfigurationGuide....................................................................................................................476.2 ServiceProviderMPLSConfigurationGuideline.............................................................................................576.3 DC1ConfigurationGuideline....................................................................................................................................626.4 ENT1ConfigurationGuide.........................................................................................................................................656.5 M1vPCScaleConfigurationGuideline.................................................................................................................68
1 Introduction The Cisco Nexus line of data center product hardware and software must pass Cisco'scomprehensive quality assurance process, which includes a multistage approach comprisingextensive unit test, feature test, and system‐level test. Each successive stage in the process addsincreasinglyhigherlevelsofcomplexityinamultidimensionalmixoffeaturesandtopologies.This document describes the NVT Phase 4.0 network topologies, hardware and softwareconfigurations,testproceduresandfindings.NVTPhase4.0testingisperformedonthefollowingnetworks:
VxLANSolution:ThistestprofileisdevelopedbasedonBGPEVPNtechnologyusingVxLANfabric. It is intend toprovidehostmobility and subnet extensionacrossdata centers andsimplify network operation using anycast gateway and does not require a first hopredundancyprotocol.
MPLSProfile1:Thisisdividedintoaccesstechnologies.o ClassicEthernetAccess:Thisnetwork focusesonbuildingandoperatinganMPLS
corenetwork thatprovidesend‐to‐endL3VPNandMVPNserviceswith theNexus7000 Sup2E andNexus 7700 Sup2E between two different sites: Site 8 ismainlyfocusedonlegacyM1moduleswhereasSite10isbuilttotestandverifytheMPLSfeaturesetover thenewF3modules.BothsitesusevirtualPort‐Channel (vPC) todeliverhighlyavailableunicastandmulticastservices.
o Fabric Path Access: In addition to the Phase 1 topology, the new Site 6 has beenintegratedtotheMPLStopology.ThisnetworkusesthenewF3moduleswithbothvirtual Port‐Channel (vPC+) and FabricPath solutions to deliver highly availableunicast and multicast services. Furthermore, the deployment of FabricPath willincreasetheoverallbandwidth(sincevPCsolutionislimitedtoonlytwovPCpeers)allowing to deploymulti‐spine network layer,while eliminating the need for STPand allowing a “plug‐and‐play” type of network as the additionof newdevices ateitherthespineortheleaflayerdoesnotpresentanysignificanttrafficlossforthepre‐establishedflows.
4
ServiceProviderMPLS:ThetopologyvalidatestheinterDCintersitedesignwithCarrierssupportingCarriersCSCbackboneforMPLSL3VPN.Twodatacenternetworksaresimulated.N7710withF3cardisusedinonedatacenterandN7010withF3isusedinthesecondnetwork.TestingisfocusedonthenetworkwithN7710.N7710andN7010withF3cardareusedasaggregationandcoreswitches.ASR9KisusedasCSCPE.N5Kisusedforlayer2accessaswellasforvPC.ASR1KisusedassecondaryRRsforredundancypurposeinthenetwork
DynamicFabricAutomation(Vinci)Profile:DFAprofileusespoweronauto‐provisioningtoconfigurenetworkdevicesandsupportdistributedgatewayfunctionandautomateddatacenter interconnect. The topology contains N7000/N7700/N6004 spines, N7000/N6001leafs, andN7004border leafs. The topology uses the latestDCNM for cablemanagementandworkflow automation. It also uses emulated leafs (Virtual ToR or VToRs) to achievescale. It supports unicast and multicast for IPv4 and IPv6 hosts and auto configurationbasedondynamicframesnoopingbasedonthetrafficandVDPrunningonthevSwitch.
DataCenter1(DC1):ThisnetworkfocusesonbuildingandoperatingadatacenterwiththeNexus7000SUP1atcoreandNexus7000SUP2ataggregationasroutingandswitchingcomponents.ItalsocoversinteroperabilitywiththeNexus5000,Nexus3000,Nexus2000,Catalyst6500/4500switches.ThisnetworkusesvirtualPort‐Channel(vPC)andFabric‐Path(vPC+)todeliverhighlyavailableunicastandmulticastservices.
ENT:ThistestprofileprimarilyisbuilttovalidateEnterprisecustomerprofiles.Inthefirstphase,wehavevalidatedtheTIERIEnterprisescustomerprofile.Thetestbedisbuiltwitha new hardware covering the existing feature set for future deployments. Nexus7000SUP2E/M2 is used at the core layer and N7700 SUP2E/F2E/F3(40G) at the aggregationlayer.TheprofilealsocoversinteroperabilitywithNexus6000andNexus3000switchesandcoversL3AggwithL3ToRandL3AggwithL2ToR.
MSDC:ThisprofilefocusesonscalerequirementsofMassivelyScalableDataCenters.Itusesafullyloaded7018peeringwithanother7018andusesF2linecardandSup2.TestsweredonewithBGPandOSPFasIGPprotocols.
M1VPC:ThistestbedfocusesonscalingthevirtualPort‐Channel(vPC]withNexus7000.Italso covers interoperability with Catalyst 6500. This network uses vPC and PVLAN todeliverhighavailabilitytoserversconnectingtodatacenters.
Operation:NetworkmanagementincludingSNMPpolingandinventorycollectionisperformedthroughDCNMfromCiscoandnetMRIfromInfoblox,TACAS+authenticationandsyslogserver.NetFlowisconfiguredtoexportthirdpartyNetflowcollectorScrutinizeroncertaintestbeds.RealhostsareconnectedbyNexusaccessswitchesusingNICteaming(inbothactivemodeandOnmode).UCS‐BseriesareconnectedtoNexusaccessswitchesthrufabricinterconnect.NTPissyncedtotheserver.
5
2 NVT Topology and Scale Achieved
2.1 VxLAN Solution Topology
Scale
Feature Scale
Maximum VLANs 40
VLANs per TOR 14
VRFs per ToR 10
Max VNI per ToR 24
Max VNI per network 10 VRFs + 30 VLANs = 40
Max underlay Multicast Groups 10 (to support broadcast + multicast traffic
Max ToR per ‘network’ 116 ( 112 ToRs per POD and 4 on peer
6
POD)
Max VTEP peers per ‘network’ 116 ( 112 ToRs per POD and 4 on peer POD) �
Overlay MAC scale 16K per POD * 2 = 32 K per ‘network’
Overlay IPv4 routes 16K per POD * 2 = 32 K per ‘network’
LISP dynamic routes 6600
Undelay v4 routes 400 across all PODs
Undelay v6 routes none
2.2 MPLS Profile 1 with F3, M1 and M2 and Fabric Path and CE Access Topology
Scale
ClassicalEthernet FabricPathFeature/Parameter Site10
(F3vPC)Site8
(M1vPC)Site6
(F3FP/vPC+)VLAN 1000 1000 1000SVI 1000 1000 1000IPv4hostsxsubnet 32 32 32HostsperVLAN 47+ 47+ 47+vPC/(vPC+) 20/na 2/na 2/(1)VLANSxvPC/(vPC+) 50/na 500/na 250/(500)FabricPathIS‐IS na na 6
7
adjacenciesPort‐channel(excludingpeer‐link,vPC&vPC+)
4 4 4
HSRPv4version2 1000 1000 1000HSRPv6version2 1000 1000 1000VRF 100 100 100VLANsperVRF 10 10 10ARPxVRF 350+ 350+ 350+mVRF 50 50 50PIMneighbordefaultVRF
4 4 4
PIMneighborxVRF 36 36 36MDTdefaulttunnel 50 50 50MDTdatatunnel 200 200 200Numberoflocalcustomergroups
10 10 10
Numberofsourcesxcustomergroup
8 8 10
NumberofOIFxgroup 10 10 10OSPFPeersdefaultVRF
4 4 4
OSPFRoutes/(Path) 48/(54) 48/(54) 48/(54)LDPSessions 4 4 4iBGPIPv4Sessions 2 2 2iBGPVPNv4Sessions 2 2 2iBGPVPNv6Sessions 2 2 2
Hardware
ClassicalEthernet‐vPC
ModelNo. Software
N7K N7KSUP2E/F3/M2/M1XL
7.2.0
C6K VS‐SUP720‐10G 151‐2.SY2CSR1000v Na 15.4.3.S1FabricPath–vPC+
ModelNo.
N7700 N7700SUP2E/F340G 7.2(0)N7000 N7KSUP1/M1 7.2(0)
8
2.3 Service Provider MPLS F3 Topology
Scale
Feature Scale
6VPEVRFs(dualstack) 100
ARPaddressesoneachswitch 21,120
GlobalMulticastroutes 100
HSRPgroups 1,000
IGMPgroups 100
L3Physicalinterfaces 300
L3VPNv4VRFs 100
MVPNVRFs 10
NumberofIPv6Host 2,000
L3Portchannel 2
9
Feature Scale
SVI 1,000
TotalMulticastprefixesinMVPN 100
TotalnumberofIPv4prefixesindefaultroutingtablewith2IGPPath 200
TotalVPNv4PrefixesfromIBGPpeerswith2IGPPath 8,000
TotalVPNv6PrefixesfromIBGPpeerswith2IGPPath 2,000
vPClinks 8
VPNv4Prefixesacross6VRFswithoneIGPPath 6,000
VPNv6Prefixesacross6VRFswithoneIGPPath 2,000
Hardware
ModelNo. ImageVersionN77K N77‐SUP2E/ N77‐
F324FQ‐257.2.0
N7K N7K‐SUP2E/ N7K‐F312FQ‐25
7.2.0
N5K N5K‐C5548UP‐SUP 7.0(5)N1(1)ASR9K ASR9001‐RP/ A9K‐MPA‐
20X1GE5.1.1
ASR1K ASR1000‐RP2/ ASR1000‐ESP20/SPA‐8X1GE‐V2
3.14
10
2.4 Dynamic Fabric Automation (Vinci) Solution Topology
Scale
Spine/SuperSpine
VRFs(vni) Transit‐Mode
SVI/BDI Transit‐Mode
VLAN/BD Transit‐Mode
BGPRoute‐ReflectorPeer 180
FabricCorePorts(Portstoleaves) 192
FabricBFDSessions 12
FabricBGPSessions 192
FabricPathLayer‐2BFD 12
Leaf FabricFacingFabricCorePorts 16FabricBFDSessions 16FabricBGPSessions 1RR VRFs(vni) 410SVI/BDI(Core) 410VLAN/BD(Core) 410
11
HostFacing VLAN(vn‐segment) 1640SVI/BDI 1640VPC+/VPC 96FEX NAPhysical/VirtualServerIPAddresses(ARP/ND)–1:6IPv4toIPv6ratio
IPv4Routes(/32+SubnetRoute) 6kIPv6Routes(/128+SubnetRoute) 4kMulticastRoutes(S,G) 2kTotalVNI/vn‐segmentperLeaf 2050
BORDERLEAF
FabricFacing FabricCorePorts 16FabricBFDSessions 16FabriciBGPSessions 1 VRFs(vni) 1000SVI/BDI(Core) 1000VLAN/BD(Core) 1000 Edge‐RouterFacing VLAN(vn‐segment) 1000Sub‐InterfaceperBorderLeaf 1000(ifVRF‐lite)
IPv4Routes(/32+SubnetRoute) 17K*
IPv6Routes(/128+SubnetRoute) 5K*MulticastRoutes(S,G) 1K* eBGP/MP‐BGPpeers 1000VPC+/VPC 0Labels/VRF OptionB
TotalVNI/vn‐segmentperBorderLeaf 1000
12
2.5 Data Center 1 Topology
Scale
Feature ScaleF1 2M1 5VDC 4FEX 6FEX‐HIF 96ACL/ACE 500POMembers 2VLAN/SVI 100VLANperFEX 10FPVLAN 50MAC 7KvPC 120VLANpervPC 1,10,16HSRPv2 100OSPFNeighbor 16BGPNeighbor 2PIMNeighbor 256PBRSequence 20UnicastRoute 10K
13
MulticastRoute 500MulticastSource 20MulticastOIF 10MulticastGroup 100FabricExtender 3FPIS‐ISadjacencies 6FPNumberofSwitchID’s 20PrimaryVLAN 20SecondaryVLAN 25Port‐channel 125Port‐channelinPVLAN 18VPCPOwithPVLAN 10Hostmode 18PromiscuousAccess 2PromiscuousTrunk 2TrunkSecondary 7PVLAN 50
2.6 Enterprise 1 Topology
Scale
14
Feature ScaleACL/ACE 500VLAN/SVI 2000MAC 1KHSRPv2 2000BGPNeighbor 18PIMNeighbor 2000PBRSequence 200UnicastRoute 22KMulticastRoute 3KMulticastGroup 390PVLAN 60vPCConfig‐Sync 120Netflow 84WCCP 58
2.7 MSDC Scale Profiles Topology
15
Scale
Parameters Profile1(L3ToR+L3Agg)
Profile2‐1(L2ToR+L3Agg)
Profile1(L3ToR+L3Agg)w/RFC5549
REQ‐ID NXOS‐MD‐OTT‐002‐001
NXOS‐MD‐OTT‐002‐003
Description F2‐Seriesbased; M2‐based;Dual‐stack;
F2‐Seriesbased;
ToR 4948EorN3K 4948Eoranyother
4948Eoranyother
Hardware *FullyloadedF2‐series
*4MSeries *FullyloadedF2‐series
SingleSup:Test1Sup2;Test2Sup1
DualSup DualSup
VDC 1VDC 1VDC 1VDCPortchannels 8‐bundlePort‐
channel8‐bundlePort‐channel
8‐bundlePort‐channel
L3ECMP Test1:16‐wayECMP;Test2:32‐wayECMP
16‐wayECMP 32‐wayECMPv4–3khostroutesanddefaultroute
UnicastRoutingProtocol
Test1‐iBGP754Adjacencies;Test2‐OSPF754Adj
OSPF,OSPFv348Adj
23iBGPv4Adjacencies[leafnodesthatdonotsupportrfc5549and16simulatedgateway/loadbalancer/firewallnodes]755iBGPv6Adjacenciesthatsupportrfc55491N7700leafnodethatsupportrfc5549
MulticastRoutingProtocols
N/A PIM‐SM,MSDPw/anycastRP
N/A
FHRP N/A 2000HSRPgroups(1000HSRPv4+1000HSRPv6)
NA
FastDetection(forIGP,PIM,FHRP)
Test1:BFD:250msx3;Test2:DefaultTimers
Test1:BFD:250msx3;Test2:DefaultTimers
Test1:BFD:250msx3;Test2:DefaultTimers
DualStack Yes Yes Yes,onsupportingleafnodesNumberof/32&/128hostentries
N/A 80Ksplit(40KIPv4+40KIPv6)
N/A
16
NumberofUnicastIGP
10KIPv4300IPv6
10K 10K
NumberofMulticastRoutes(S,G)
N/A 15K N/A
VLANs N/A 250 N/A
SVI(sameasVLANs)
N/A 250 N/A
ACLs N/A 250 N/A
ARP/NDP(30secondsARPrefreshrate)
N/A 80K(40KARPand40KNDP)
N/A
ARP/NDPLearningRate
N/A 5000pps N/A
ARP/NDPGleanrate
N/A 1000pps N/A
DHCPRelay N/A 100pps N/AMIB KPIX KPIX KPIX
2.8 M1 vPC Scale
FeatureMax M1vPCPrimaryVLAN 140SecondaryVLAN 280Phy.PortsusedforPVLAN 468Port‐channel 388Port‐channelinPVLAN 388VPCPOwithPVLAN 256Hostmode 32PromiscuousAccess 64PromiscuousTrunk 186TrunkSecondary 186
17
Topology
3 ISSU Matrix Image ISSU/COLDBOOT MPLS
Profile1
DC1 ENT MSDC M1vPC
7.2.0>7.2.0.upg ISSU PASS PASS PASS PASS PASS6.2.12>7.2.0 ISSU PASS PASS PASS PASS 6.2.10>7.2.0 ISSU PASS PASS PASS 6.2.8a>7.2.0 ISSU PASS 6.2.8b>7.2.0 ISSU PASS PASS 7.2.0>7.2.0.upg COLDBOOT PASS 6.2.12>7.2.0 COLDBOOT PASS PASS 6.2.10>7.2.0 COLDBOOT PASS PASS 6.2.8a>7.2.0 COLDBOOT PASS 6.2.8b>7.2.0 COLDBOOT PASS PASS PASS PASS7.2.0>6.2.12 COLDBOOT PASS PASS 7.2.0>6.2.10 COLDBOOT PASS 7.2.0>6.2.8a COLDBOOT PASS 7.2.0>6.2.8b COLDBOOT PASS PASS
18
4 Profile Details
4.1 VxLAN Solution
4.1.1 Network Topology and Design Review TheDCdesignisstructuredinpairsofPODsbetweenwhichmobilityandsegmentationshouldbesupported.ThechoiceistouseVXLANwithanEVPNcontrolplanewithineachPODandforeast‐westconnectivityacrosspeerPODs;LISPisrequiredforoptimizationofNorth‐Southtraffic.Thereare4aggregationboxesperPODthatuseSUP2EandF3linecardsand40GEinterfaces.EachaggregationboxisconnectedtoallToRs.The4aggregationboxesarealsoconnectedtoeachotherinaring.EachRackhastwoToRsinavPCarray.vPCrunsfromthephysicalhoststotheToRs.CommunicationbetweenPODsinthesameDChappensoverthecore.North‐SouthCommunicationinandoutofthePODsalsohappensoverthecore.Segmentationisonlystretchedbetweenpeer‐PODs;segmentationisnotstretchedoutsideofthePODstotheCore.OnlyoneVRF(theproductionVRF)isroutableoverthecore.Thus,inter‐PODandNorth‐SouthcommunicationislimitedtotheproductionVRFonly.East‐WestcommunicationisoptimizedbetweenpairsofPODsacrossDCs.EachPODwillhaveapeerPODattheremoteDCandtheaggregationswitchesofthesepeerPODswillbedirectlyconnectedoverfiber.Thefollowingconnectionswillbeestablished:POD1toPOD1’East‐WestcommunicationsbetweenPeerPODswillfollowthelowlatency/high‐capacitydirectconnectionbetweenAggregationswitches.CommunicationacrossPODsthatarenotpeerswillhappenoverthecoreonlyforthe“ProductionVRF”DesignChoices
Spine to Core is eBGP, peering is interface based. On Spine only Production_vrf routes are routable in core
Core is single AS OSPF will be running between all Spines in the ring topology Spines are iBGP peering over interface to TORs. Loopback are redistributed with network
statement. All four spines are IPv4 RR to all ToRs Each N9396 are connected to all four Spines. Connection is 40G links. Some connection uses are
port‐channel with 2‐3 40G member link Legacy TORs has 10 VRFs. Correspondingly Spine will have 10 VRF‐lite for Intra‐POD legacy to
fabric traffic Legacy TORs eBGP peer to Spine thru "fake iBGP" by manipulating "local AS" in as‐path. So BGP
peering is iBGP, but local AS are different so route get redistributed to MP‐iBGP L2VPN EVPN.
Spines are route reflector for IPv4 and L2VPN EVPN
All leafs in the POD are EVPN RR clients to Spins. IBGP peering is with loopback
Total 10 VRF in system, one is Production VRF which will carry hosts that will require inter‐POD
and N‐S (i.e branch to host) access
19
Other 9 VRFs will be connecting within the POD or peer‐POD only. No N‐S access for these VRFs.
There is no route‐leaking into Production VRF. These 9 VRFs require connectivity to legacy ToRs
also
VM hosts requiring mobility are separated into different VLANs (we call it mobility VLAN)
Production VRF will have 10 Mobility Vlan with total of 6600 host and 500 Non‐mobility vlans
with 4 Vlan each in each TORs (125 TORs X 4 Vlans = 500)
Each Mobility Vlan will have 3 host (10 Vlan x 3 host X 125 Racks = 3750 hosts) in POD A and
2850 hosts in POD A”
For Non Mobility Vlans, 4 Vlan X 50 host X 125 TORs = 25000 hosts in POD A
Inter‐peer‐POD is EBGP interface peering over high speed direct link. VXLAN tunnels are TOR to
TOR
For N‐S for Production VRF, VXLAN tunnel will terminate on Spine and L3VRF handoff to
production‐vrf handoff
IPv4 and L2VPN EVPN iBGP peering between inter‐spine ring links and Set local pref low for inter
spine link
POD A has 250 emulated TORs using IXIA eVPN emulation
Each TORs N9K have 40 host VPCs with N3K fan out switches
10 Legacy TORs are connected to PODA
2 map server (CSR1KV) connected with DDT to Branch router ASR1K.
LISP traffic will load balance between 4 spines by the map‐server
Spine we have configured route-map to allow only non-mobility prefixes to get advertise to core and branch for south to north traffic
2 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
BGP Design
N9300 N9300 N9300 N9300
N7700 N7700
N3000 N3000
N9300 N9300 N9300 N9300
N7700 N7700
N3000 N3000
AS 100 local as 300 remote-as 200 prepend
AS 100
“fake” eBGP ipv4
eBGP pfx list allow non-mobility pfxs and legacy pfxs
IGP
iBGP lo0 Ipv4, evpn local_pref 50
iBGP ipv4 intf RR clients each other local_pref 50 next-hop-peer-addr
iBGP ipv4 intf iBGP evpn lo0 Spine: next-hop-peer-addr
eBGP intf ipv4 lo0 evpn
AS 100 local as 200 remote-as 300 prepend
20
3 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
LISP
N9300 N9300 N9300 N9300
N7700 N7700
N3000 N3000
N9300 N9300 N9300 N9300
N7700 N7700
N3000 N3000
Branch MS
MS
ASR1K
1
2
3a
3b 5
6
4
7
1. GARP 2. BGP EVPN update (MAC, IP) 3. a. eBGP EVPN update 3. b. LISP update MS 4. Branch XTR send traffic to XTR old POD 5. LISP SMR to Branch XTR to check MS 6. Branch XTR query MS 7. Traffic send to new POD XTR
4 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Data Packet Flow
N9300 N9300 N9300 N9300
N7700 N7700
N3000 N3000
N9300 N9300 N9300 N9300
N7700 N7700
N3000 N3000
MS
MS
ASR1K
Native IPv4
VxLAN Encaped
LISP Encaped
21
4.1.1.1 VxLAN Solution Topology
4.1.2 Hardware and Software Overview
ModelNo. NVT4.0N77K N77‐SUP2E/N77‐
F324FQ‐257.2.0
N9K N9K‐C9396PX/N9K‐M12PQ
7.0.3
CSR1000v Na 15.4(3)S1ASR1K ASR1000‐RP2/
ASR1000‐SIP10/SPA‐10X1GE‐V2
155‐1.S1
ASR9K A9K‐RSP440‐TR/A9K‐40GE‐E/A9K‐2x100GE‐SE
5.3.0
4.1.3 VxLAN Solution Configuration Guide Pleaseseetheappendixforconfigurationdetails
4.1.4 VxLAN Solution Scale
Feature Scale
Maximum VLANs 510
VLANs per TOR 14
VRFs per ToR 10
Max VNI per ToR 24
Max VNI per network 10 VRFs + 500 VLANs = 510
22
Max underlay Multicast Groups 510 (to support broadcast + multicast traffic
Max ToR per ‘network’ 250�(125 ToRs per POD * 2) = We build in pairs.
Max VTEP peers per ‘network’ 250�(125 VTEPs per POD * 2) = We build in pairs.
Overlay MAC scale 16K per POD * 2 = 32 K per ‘network’
Overlay IPv4 routes 16K per POD * 2 = 32 K per ‘network’
LISP dynamic routes 6600
Undelay v4 routes 400 across all PODs
Undelay v6 routes none
4.2 MPLS Profile 1 with F3, M1 and M2 and Fabric Path and Classical Ethernet Access TheMPLSprofileisperformedintwoaccessmethodtechnologies.
ClassicalEthernet:Corenetwork,Site8andSite10.Thisnetwork focusesonbuildingandoperatinganMPLScorenetworkthatprovidesend‐to‐endL3VPNandMVPNserviceswiththeNexus7000Sup2EandNexus7700Sup2Ebetweentwodifferentsites:Site8ismainlyfocusedonlegacyM1modules,whereasSite10isbuilttotestandverifyMPLSfeaturesetover the new F3 modules. Both sites use virtual Port‐Channel (vPC) to deliver highlyavailableunicastandmulticastservices.
FabricPath: In addition to theabove topology, thenewSite6hasbeen integrated to theMPLS topology. This network uses the new F3 modules with both virtual Port‐Channel(vPC+)andFabricPathsolutionstodeliverhighlyavailableunicastandmulticastservices.Furthermore,thedeploymentofFabricPathwillincreasetheoverallbandwidth(sincevPCsolution is limited to only two vPC peers) allowing to deploymulti‐spine network layer,whileeliminatingtheneed forSTPandallowinga“plug‐and‐play” typeofnetworkas theadditionofnewdevicesateitherthespineoftheleaflayerdoesnotpresentanysignificanttrafficlossforthepre‐establishedflows.
4.2.1 MPLS Topology Design Review Thetopologiesandtestcasesvalidatehigh‐availabledatacenternetworksinordertoprovideunifiedfabricandcomputingservices.ThisisachievedbyusingtheNexus7000andN7700,withfeaturessuchasvPCwithdifferentNexus7000modules.Core Network ThecorenetworkisbuiltaroundtwoNexus7000chassiswithSup2E.EachchassisisconfiguredtohavetwoProutersintwodifferentVDCs(foratotaloffourProuters).
Nexus7000withdistributedport‐channelsconnectedinasquaretopology CSR1kvareconfiguredasaroute‐reflectorclusterhenceconnectedtotwooftheoppositeP
routers(P1andP3)Site 10 Network – Classical Ethernet F3 ThissiteisbuiltaroundtheNexus7700withSup2EwithF3modules:
23
TwoNexus7700withvPCactingasredundantmulti‐homingPEs,whichareconnectedtotwootherNexus7700foraggregation/accessactingasCEsdevicesconfiguredondifferentVDCs.
Site 8 Network – Classical Ethernet M1 ThissiteisbuiltaroundtheNexus7000withSup2EwithM1andM1XLmodules:
TwoNexus7700withvPCactingasredundantmulti‐homingPEswhichareconnectedtotwoCatalyst6500foraggregation/accessactingasCEsdevices
WhilethemajorityoftestcasesfocusonintegratedsolutionsusingNexusswitching,modularCatalystswitchesarealsoincludedforinteroperabilitybetweenNX‐OSandIOS.Site 6 Network – Fabric Path Thetopologiesandtestcasesvalidatehigh‐availabledatacenternetworksinordertoprovideunifiedfabricandcomputingservices.ThisisachievedbyusingtheNexusN7700,withfeaturessuchasFabricPathandvPC+withF3/40GNexusmodulesInFigure1,Site6isthesitethathadbeendeployedtoverifyandtesttheFabricPathfeaturesetontheNexus7700withSup2EandF3/40Gmodules.
Nexus7700/F3‐40GmoduleswithvPC+toNexus7000/M1modules. Nexus7700/F3‐40GmoduleswithvPCtoNexus7700/F3‐40Gmodulesfor
FabricPathwithdifferenttopologies.
4.2.2 Hardware and Software Overview
ClassicalEthernet‐vPC
ModelNo. Software
N7K N7KSUP2E/F3/M2/M1XL
7.2.0
C6K VS‐SUP720‐10G 151‐2.SY2CSR1000v Na 15.4.3.S1FabricPath–vPC+
ModelNo.
N7700 N7700SUP2E/F340G 7.2(0)N7000 N7KSUP1/M1 7.2(0)
24
4.2.3 Topology
4.2.4 SP MPLS Scale Coverage
Phase1 Phase2Feature/Parameter Site10
(F3vPC)Site8
(M1vPC)Site6
(F3FP/vPC+)VLAN 1000 1000 1000SVI 1000 1000 1000IPv4hostsxsubnet 32 32 32HostsperVLAN 47+ 47+ 47+vPC/(vPC+) 20/na 2/na 2/(1)VLANSxvPC/(vPC+) 50/na 500/na 250/(500)FabricPathIS‐ISadjacencies
na na 6
Port‐channel(excludingpeer‐link,vPC&vPC+)
4 4 4
HSRPv4version2 1000 1000 1000HSRPv6version2 1000 1000 1000VRF 100 100 100VLANsperVRF 10 10 10ARPxVRF 350+ 350+ 350+mVRF 50 50 50
25
PIMneighbordefaultVRF
4 4 4
PIMneighborxVRF 36 36 36MDTdefaulttunnel 50 50 50MDTdatatunnel 200 200 200Numberoflocalcustomergroups
10 10 10
Numberofsourcesxcustomergroup
8 8 10
NumberofOIFxgroup 10 10 10OSPFPeersdefaultVRF
4 4 4
OSPFRoutes/(Path) 48/(54) 48/(54) 48/(54)LDPSessions 4 4 4iBGPIPv4Sessions 2 2 2iBGPVPNv4Sessions 2 2 2iBGPVPNv6Sessions 2 2 2
4.2.5 SP MPLS ISSU Matrix
ISSU/Coldboot (SUP2E)ISSU6.2.10‐>7.2.1 PassColdboot6.2.10‐>7.2.1 PassISSU6.2.8b‐>7.2.1 PassColdboot6.2.8b‐>7.2.1 PassISSU6.2.12‐>7.2.1 PassColdboot6.2.12‐>7.2.1 Pass
4.3 SP MPLS Solution
4.3.1 Network Logical Topology Design Overview MPLSSolution:ThefocusofthistestbedistotesttheMPLSsolutionproposedtothedatacentercustomersingeneral.Scaleandtopologyarealignedtothedeploymentproposedtothetoaspecificcustomer.SolutiontestingisdoneforinterDC(layer3DCIwithMPLS)withCSCbackbone.TestingisprimarilyfocusedonMPLSfeaturesconfiguredonN77KwithF3cardpositionedasaggregationandcoreswitchesinthedatacenternetwork.ThisnetworkusesvPCtowardsaggregationtodeliverhighlyavailableunicastandmulticastservicesandCSCintheMPLSbackbone.ASR9KsareusedasCSCPEs.TheoverallgoalistoidentifyanystabilityissueswhenconfiguredwithmultiplefeaturesandmeasuretrafficconvergencemetricwithkeytriggerssuchasSSO,processcrashes/restarts,VDCreloadandOIRwithmultiplefeaturesconfiguredatpre‐definedscales.Theresultsinthisdocumentarecloselytiedwiththesetuputilizedforthisprofile.Theseresultsmayvarydependinguponthedeploymentandoptimizationstunedfortheconvergence.
26
ThetopologyvalidatestheinterDCintersitedesignwithCSCbackboneforMPLSL3VPN.Twodatacenternetworksaresimulated.N7710withF3cardisusedinonedatacenterandN7010withF3isusedinthesecondnetwork.TestingisfocusedonthenetworkwithN7710.N7710andN7010withF3cardareusedasaggregationandcoreswitches.ASR9KisusedasCSCPE.N5Kisusedforlayer2accessaswellasforvPC.ASR1KisusedassecondaryRRsforredundancypurposeinthenetworkasshowninFigure1.
4.3.2 Description of the test network
N7710andN7010withF3cardareusedasaggregationandcoreswitches‐FeaturesconfiguredareL3VPNwithVRFs,ISIS,iBGP,HSRP,SVIs,vPC,LDP,6vPE,mvpn,multicast.
ASR9KisusedasCSCPE–FeaturesconfiguredareOSPF,iBGP,LDP,VRF. N5Kisusedforlayer2accessaswellasforvPC. ASR1KisusedassecondaryRRsforredundancypurposeinthenetwork‐Features
configuredareOSPF,iBGP,LDP Ixiaisusedtogeneratehostroutes(/32)aswellforendtoendtrafficvalidation.
4.3.3 Hardware and Software Overview ModelNo. ImageVersion
N77K N77‐SUP2E/ N77‐F324FQ‐25
7.2.0
N7K N7K‐SUP2E/ N7K‐F312FQ‐25
7.2.0
N5K N5K‐C5548UP‐SUP 7.0(5)N1(1)ASR9K ASR9001‐RP/ A9K‐MPA‐
20X1GE5.1.1
ASR1K ASR1000‐RP2/ ASR1000‐ESP20/SPA‐8X1GE‐V2
3.14
27
4.3.4 Topology
4.3.5 Scale ThescalenumberslistedbelowareconfiguredintheN77K‐PE1andN77K‐PE2(asshownintheTopologyabove)Aggregationswitches.Feature Scale
6VPEVRFs(dualstack) 100
ARPaddressesoneachswitch 21,120
GlobalMulticastroutes 100
HSRPgroups 1,000
IGMPgroups 100
L3Physicalinterfaces 300
L3VPNv4VRFs 100
MVPNVRFs 10
NumberofIPv6Host 2,000
28
Feature Scale
L3Portchannel 2
SVI 1,000
TotalMulticastprefixesinMVPN 100
TotalnumberofIPv4prefixesindefaultroutingtablewith2IGPPath 200
TotalVPNv4PrefixesfromIBGPpeerswith2IGPPath 8,000
TotalVPNv6PrefixesfromIBGPpeerswith2IGPPath 2,000
vPClinks 8
VPNv4Prefixesacross6VRFswithoneIGPPath 6,000
VPNv6Prefixesacross6VRFswithoneIGPPath 2,000
4.3.6 ISSU Matrix Sincethisisanewdeployment,ISSUfrom7.2to7.2upgradeimageswastested
4.3.7 Traffic and Convergence Asmentionedpreviously,theoverallgoalofthistestingistocheckthestabilityoftheN77KasaggregationandRRcuminlinecoreandmeasurethetrafficconvergenceduringvariousoperationaltriggers.Thetrafficconvergencemeasurementsareintheattachedspreadsheet.AllthetrafficconvergencemeasurementsarefocusedonN77KusedaseitheraggregationswitchorN77KasRRcuminlinecore.Thetrafficconvergencemetricsforeachtriggerarenotconsistenteverytime.Soeachtriggerisrepeated3to4timesandthemaximumvalueobservedduringanyoftheiterationisrecordedasshowninthetablebelow.ThefollowingtrafficflowmetricsareconfiguredandtestedusingIxia.Fixedframesizeof100isusedforallstreams.TrafficStreams Description TrafficRateV4trafficbethostsondiffDCs
BidirectionalFlowbetween900v4hosts
100KFrames/Sec
V6trafficbethostsondiffDCs
BidirectionalFlowbetween200v6hosts
100KFrames/Sec
V4Scaletraffic Bidirectionalflowforrestofthev4prefixesfromremoteDCforscaleasspecifiedinSection2
100KFrames/Sec
V6ScaleTraffic Bidirectionalflowforrestofthev6prefixes
100KFrames/Sec
29
fromremoteDCforscaleasspecifiedinSection2
Multicast(mvpn)withN77KconnectedtoReceiversonIxia
100flowsfor100groups
10KFrames/Sec
Multicast(mvpn)withN77KconnectedtoSourceonIxia
100flowsfor100groups
10KFrames/Sec
GlobalMulticastwithN77KconnectedtoReceiversonIxia
100flowsfor100groups
10KFrames/Sec
GlobalMulticastwithN77KconnectedtoSourceonIxia
100flowsfor100groups
10KFrames/Sec
V4InterVRFtraffic IntervrftrafficgoingthroughfirewallwithintheDC
200KFrames/Sec
V6InterVRFTraffic IntervrftrafficgoingthroughfirewallwithintheDC
200KFrames/Sec
CPUandmemoryutilizationontheN77K‐PE1andN77K‐PE2measuredundersteadystateareshownbelow.Thesteadystatemeasurementsaredonewhenthesystemisstablewithallthefeaturesconfiguredand100%endtoendtrafficrunning.SNMPpollingofMPLSMIBsaredonecontinuouslyinthebackground.Device AverageCPUUtilization
(5minwith16coreCPU)MemoryUtilization
N77K‐PE1 1.49 6738104KN77K‐PE2 0.51 7957180KDuetotheredistributionoftheISISroutesintoOSPFandvice‐versatowardstheCSCbackbone,routingloopsareboundtohappenduringcertaintriggerslikelinkfailure,vdcreloadetc.Toavoidthis,itisrecommendedtohaveroute‐mapsattachedsuchthat,forlocallylearntroutesISIShashigherpriorityandforremoteroutes,OSPFhashigherpriority.
4.4 Dynamic Fabric Automation (Vinci)
4.4.1 Topology Design Review Thereare9Spines,180Leafsand2BorderLeafsinthistopology.SpineS1andRemoteDC‐SpineareNexus6004Switches.SpineS8isNexus7710ChassiswithF3CardandSpineS2
30
toS7areVDCswithinNexus7009ChassiswithF2eCards.LeafsL174toL177areNexus6001Switches.LeafsL178andLeaf179areNexus7009ChassiswithF3CardsandLeaf0toLeaf173areemulatedusingTitanium+VTORs.Border39andBorderLeaf40areNexus7706andNexus7710ChassisrespectivelywithF3cards.FabricConnectivityRealLeafs(L174‐L179)andBorderLeafs(BL38‐BL40)areconnectedto8spines.Spine8isconnectedto174EmulatedLeafs(L0‐L173)withthehelpofFanoutswitch(Nexus7018).UCSsusedforemulatingVTORsareconnectedtoFanoutswitchwithaTrunkLinkasdescribedinVinciHybridClusterapproach.ServerLeafConnectivityForconfiguringVM,weuseEmulatedhosts.IXIAtrafficgeneratorwillbeusedforemulatinghostwithVDPandARP/NDSupportconnectedtoRealLeafs.DCIandL3VPNConnectivityatBorderLeafBGPbasedVRFLitewillbeconfiguredasborderLeafnodeandDCInode.BGPMPLSVPNwillbeconfiguredbetweenDCIPeers.IXIAemulationwillbeusedtoemulateremotebranchroutes.
4.4.2 Hardware and Software Overview ModelNo. ImageVersion
N77K N7F‐SUP2E /N77‐F324FQ‐25/N77‐F248XP‐23E/N7F‐F248XP‐23E
7.2(0)D1(1)
N7K N7K‐SUP2E / N7K‐F312FQ‐25/N7K‐F248XP‐25E/N7K‐F248XP‐25
7.2(0)D1(1)
N6K N6K‐C6001‐64P‐SUP /N6K‐C6001‐M4Q
7.1(1)N1(1)
N6K N6K‐C6004‐96Q‐SUP /N6K‐FIXED‐LEM/N6K‐C6004‐M12Q
7.1(0)N1(1)
N6K N6K‐C6004‐96Q‐SUP /N6K‐FIXED‐LEM
7.0(2)N1(1)
CiscoPrime DataCenterNetworkManager
7.2(0.14)
31
4.4.3 Topology
4.4.4 Traffic and Metrics
IPv4 Unicast Traffic Across Different Leaf/Across Border Leaf
IPv6 Unicast Traffic Across Different Leaf/Across Border Leaf
IPv4 Multicast Traffic Across Different Leaf/Across Border Leaf
Device AverageCPU
Utilization/LoadAverage15mins
MemoryUtilization
Leaf–PrimaryvPC 0.34 Memoryusage:5965956Kused
Leaf–SecondaryvPC 0.25 Memoryusage:5967084Kused
Spine 0.36 Memoryusage:5757104Kused
BL 0.43 Memoryusage:6308256Kused
32
4.5 DC1
4.5.1 Hardware and Software Overview
Platform ModelNo. SoftwareN7K N7KSUP2/F1/M1 7.2.0N5K N5K‐C5548UP‐SUP 7.0.6.N1.1N3K N3K‐C3048TP‐1GE‐SUP 5.0.3.U5.1cC6K VS‐SUP720‐10G 151‐1.SY WS‐SUP720 122‐33.SXJ4C4K WS‐X45‐SUP7‐E 03.03.02.SG.151‐1.SG2 WS‐C4948 150‐2.SG6‐6.10N2K C2224TP;C2248TP;C2232PP
4.5.2 Network Logical Topology Design Overview
Thetopologiesandtestcasesvalidatehigh‐availabledatacenternetworksinordertoprovideunifiedfabricandcomputingservices.ThisisachievedbyusingtheNexus7010andNexus5548withfeaturessuchasvPCandFabric‐Path.Nexus7010areinstalledwithF1andM1linecardstoprovidelegacyL2&L3andvPClegportchannelconnectivitywithpeerdevicesandF1portsareusedforfabric‐pathport‐channelwithN5000peers.ArrayofNexus2000areconnectedwithNexus7010asfabric‐extenders(FEX).AccesslayerswitchesinthisdatacenterextendedtoIXIA(TrafficGenerator)portstosimulateendhosts/serverstosendandreceiveunicast&multicasttrafficThedatacentersiteisbuiltaroundtheNexus7010SUP2ataggregation.Thisdatacentersiteissplitintotwohalves:DC101:(vPC)
Nexus7000withvPCtoNexus5000,C4K/C6Kforaccess Nexus7000withlegacyether‐channel(trunk)withC4K Nexus7000withNexus2000FEX Nexus7000withL3toNexus3048
DC102:(vPC+)
Nexus7000(spine)withFabric‐PathtoNexus5000(leaf) Nexus7000withvPCtoC6Kforaccess Nexus7000withL3toNexus3048
33
DC101Device Platform Position VPCStatusDC101‐5(VPCSec.) N7K Aggregation VPCDC101‐6(VPCPri.) N7K Aggregation VPCDC101‐7 C6K Access VPCDC101‐8 C6K Access VPCDC101‐17(VPCPri.) N5K Access DualsidedVPCDC101‐18(VPCSec.) N5K Access DualsidedVPCDC101‐27(VPCPri.) N5K Access DualsidedVPCDC101‐28(VPCSec.) N5K Access DualsidedVPCDC101‐37 C4K Access NonVPCDC101‐38 C4K Access NonVPCFEX101,102,103 N2K AccessFEX VPC‐HIFDC101‐47 N5K Access L3
DC102Device Platform Position VPCStatusDC102‐51(VPC+Sec.) N7K Aggregation Fabric‐path‐SpineDC102‐52(VPC+Pri.) N7K Aggregation Fabric‐path‐SpineDC102‐53(VPC+Sec.) N7K Aggregation Fabric‐path‐SpineDC102‐54(VPC+Pri.) N7K Aggregation Fabric‐path‐SpineDC102‐701(VPC+Pri.) N5K Access Fabric‐path‐LeafDC102‐702(VPC+Sec.) N5K Access Fabric‐path‐LeafDC102‐703(VPC+Pri.) N5K Access Fabric‐path‐LeafDC102‐704(VPC+Sec.) N5K Access Fabric‐path‐LeafDC102‐705(VPC+Pri.) N5K Access Fabric‐path‐LeafDC102‐706(VPC+Sec.) N5K Access Fabric‐path‐LeafDC102‐7011(VPC+Pri.) N5K Access DualsidedVPCDC102‐7012(VPC+Sec.) N5K Access DualsidedVPCDC102‐17 C6K Access VPCDC102‐18 C6K Access VPCDC102‐27 C6K Access VPCDC102‐47 N3K Access L3
4.5.3 Configuration Details
Thefollowingconfigurationsareappliedtothetestnetwork:
34
Commonsystemcontrol,managementandaccounting:CommonsystemfeatureslikeSSH,TACACS+,Syslog,SNMP,NTP,SPAN,DNSandManagementVRFareconfigured
BGP:eBGPisconfiguredbetweenthecoreswitchesandthepubliccloud.
OSPF:OSPFistheIGPrunningacrossthenetwork.Eachaggregation‐accessblockisconfiguredasauniqueareawiththecoreswitchesplayingtheroleoftheABR.
PIM‐SM:PIMSparseMode/PIMAnySourceMulticastisdeployedacrossthenetworktosupportmulticast.Eachaggregation‐accessblockisconfiguredwiththeRPforlocallysourcedgroups.
MSDPAnycastRP:MSDPisdeployedtoexchangesourceinformationbetweenAnycastRPs.
vPC:vPCtechnologyisdeployedintheaggregation‐accessblockDC101asshowninFigure1.Inaddition,dual‐sidedvPCisconfiguredbetweentheNexus7000andNexus5000switches
FP:FabricPathisdeployedintheaggregationblockDC102.Thespinelayercomprises
Nexus7000switchesandtheleafswitchesaredeployedusingNexus5000switches.
VLANtrunking:VLANtrunkingisusedintheaggregation‐accessblockstomaintainsegregationandsecurity.
STP:RapidSpanningTreeProtocolisusedtopreventLayer2loopsintheaggregation‐
accessblocks.Thespanningtreerootisplacedontheaggregationlevel.RootGuardisconfiguredontheaggregationleveltoenforcerootplacement.BPDUFilter,BPDUGuardandPort‐FastEdgeareconfiguredontheaccessportstowardshosts.
HSRP:HSRPv2isusedasthefirsthopgatewayprotocolforhosts;HSRPconfiguredatN7KaggregationVDCs(DC101‐5,DC101‐6&DC102‐51,DC102‐52,DC102‐53,DC102‐54)
FEX:ThreesetofFabricExtenders(Nexus2000)FEX101,FEX103&FEX104aredeployed
onNexus7000atDC101
FEXuplinksarelegacyportchannelswhereasFEXhostportsareconfiguredasvPC.
IGMP:IGMPisusedbyhoststojoinmulticastgroupsofinterest.IGMPsnoopingisenabledonallswitchesintheaggregation‐accessblockstopreventfloodingofmulticastdatatraffic.
LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork.
UDLD:UDLDaggressivemodeisconfiguredacrossthenetworktodetectandprevent
unidirectionallinks
PVLAN:Promiscuous/Isolated/CommunityPVLANconfiguredbetweenNexus7000toNexus5000,C6KandC4KovervPCaswellasnonVPCatDC101.PVLANtrafficconfiguredoverFabric‐pathPOatDC102
35
4.5.3.1 DC1 Topology
36
4.6 ENT
4.6.1 Hardware and Software Overview Platform ModelNo. SoftwareN7000 SUP2E,M2 7.2.0N7700 SUP2E,F2E,F3(40Gwith
breakout)7.2.0
N6000 N6K‐C6004‐96Q‐SUP,N6K‐C6001‐64P‐SUP
7.1.0.N1
N3000 N3K‐C3548P‐10G‐SUP,N3K‐C3048TP‐1GE‐SUP
6.0.2
4.6.2 Network Logical Topology Design Overview Thetopologiesandtestcasesvalidatehigh‐availabledatacenternetworksinordertoprovideunifiedfabricandcomputingservices.ThisisachievedbyusingtheNexus7010andNexus7700withfeaturessuchasvPCscale,VRF,PVLAN,ACL,netflow,wccp,multicast,dualvpc,etc.Nexus7710installedwithF2EandF3linecardsprovidelegacyL2&L3andvPClegportchannelconnectivitywithpeerdevices.AccesslayerswitchesinthissetupareextendedtoIXIA(TrafficGenerator)portstosimulateendhosts/serverstosendandreceiveunicast&multicasttrafficThedatacentersiteisbuiltaroundtheNexus7010withSUP2Eatcore&Nexus7710withSUP2Eataggregation.ENT1Aggregation:Nexus7710with120VPCPOtoN6004Nexus7710withDualVPCPOtoN6001Nexus7710withVRFsonF2physicalportsand6ECMPtoN3kNexus7710withVRFsonF3Port‐channels/sub‐interfacetoN3kNexus7710withPVLANonF3andF2toN3K
ENT1Device Platform Position StatusEnt‐1(VDC) N7000 Backbone L3Ent‐3 N7000 Core L3Ent‐4 N7000 Core L3Ent‐5 N7700 Aggregation VPCEnt‐6 N7700 Aggregation VPC
37
VPC‐sim1 N6004 VPCsimulator FanoutVPC‐sim2 N6004 VPCsimulator FanoutVPC‐sim3 N3548 Access No‐VPC/RegularL2Ent‐701 N6001 Access DualsidedVPCEnt‐702 N6001 Access DualsidedVPCEnt‐703 N3048 Access L3withVRFEnt‐704 N3048 Access L3withVRFEnt‐705 N3048 Access L3withVRFEnt‐706 N3048 Access L3withVRFEnt‐FO1 N3048 Fanout Fanout/RegularL2
4.7 Configuration Details Thefollowingconfigurationsareappliedtothetestnetwork:
Commonsystemcontrol,managementandaccounting:CommonsystemfeatureslikeSSH,TACACS+,Syslog,SNMP,NTP,SPAN,DNSandManagementVRFareconfigured
BGP:eBGPisconfiguredbetweenthecoreswitchesandthepubliccloud.
PIM‐SM:PIMSparseMode/PIMAnySourceMulticastisdeployedacrossthenetworktosupportmulticast.Eachaggregation‐accessblockisconfiguredwiththeRPforthelocallysourcedgroups.
MSDPAnycastRP:MSDPisdeployedtoexchangesourceinformationbetweenAnycastRPs.
vPC:vPCtechnologyisdeployedintheaggregation‐accessasshowninFigure1.Inaddition,dual‐sidedvPCisconfiguredbetweentheNexus7000andNexus6000switches
STP:RapidSpanningTreeProtocolisusedtopreventLayer2loopsintheaggregation‐
accessblocks.Thespanningtreerootisplacedontheaggregationlevel.RootGuardisconfiguredontheaggregationleveltoenforcerootplacement.BPDUFilter,BPDUGuardandPort‐FastEdgeareconfiguredontheaccessportstowardshosts.
HSRP:HSRPv2isusedasthefirsthopgatewayprotocolforhosts;HSRPconfiguredatN7Kaggregation
IGMP:IGMPisusedbyhoststojoinmulticastgroupsofinterest.IGMPsnoopingisenabledonallswitchesintheaggregation‐accessblockstopreventfloodingofmulticastdatatraffic.
LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork.
38
4.8 ENT Topology
39
4.9 MSDC Scale ThetopologiesandtestcasesvalidatetherequirementoffeatureandscaleinaMassiveScalableDatacenter(MSDC)deployment.ThisisachievedbyusingtheNexus7010,Nexus3k,andNexus5548withprotocolssuchasOSPF,BGP,andBFDtocreatevariousprofiles.Thefollowingfeaturesarecoveredinprofile1:‐ BGPorOSPF BFDwithdefaulttimers(50ms)or250ms SNMP NTP SYSLOG
Thefollowingfeaturesarecoveredinprofile2‐1: OSPF/OSPFv3 BFD(v4andv6)withtimersof250ms vPC MulticastwithMSDP(v4) IGMP ACL DHCPrelay Highavailability SNMP NTP SYSLOG
4.9.1 Topology Design Overview MSDCcovered2mainprofiles–profile1withBGP/OSPFandprofile2‐1
1) Profile1–leafnodeona7018with16F2,runningeitherBGPorOSPFasroutingprotocolwithBFDusingdifferentbfdtimers,default(50ms)or250ms.
2) Profile2‐1–L2/L3ona7018with2M2,runningospf/ospfv3asroutingprotocolwithBFDusing250mswithvpc,multicast.
RolesandRouters/devicesRoles RoutersandDevices
Spinenode N7K‐114‐101(UUT1),N7K‐114‐102defaultVDC(UUT2)andVDC1andN7k‐114‐108VDC1.16/32ECMPisbetweenspinenodeandgateway/loadbalancersandfirewall.
Leafnode/ToR N3Kandsimulatedby4510‐1‐fanout/4510‐2‐fanout/4510‐3fanout+Ixia
Gateway,Loadbalancers,Firewall
SimulatedbyN7K‐114‐108defaultVDCand4507‐1‐fanout+Ixia
40
4.9.2 ISSU Matrix
ISSU/ColdbootISSU6.2.8a‐>7.2ISSU6.2.10‐>7.2 PassISSU6.2.12‐>7.2 PassISSU7.2‐>upg Pass
4.10 M1 vPC Scale
4.10.1 Hardware and Software Overview
Platform ModelNo. SoftwareN7K N7KSUP2EM1 7.2.0.C6K VS‐SUP720‐10G 122‐33.SXJ1
4.10.2 Network Logical Topology Design Overview
41
Thetopologyvalidateshigh‐availablenetworksthatdepictthevariousprivateVLANfeatureimplementationsinordertoprovideanideaoftheprivateVLANscalenumberssupported.ThisisachievedbyusingtheNexus7000andCatalyst6500switches.Figure 3 illustrates thenetworkbuilt around2Nexus 7000 switcheswith Sup2e andM1modules.Thetopologycontains:
Nexus7000withvPCtoCatalyst6500VSSswitchesforaccessconfigured. Nexus7000connectedtoCatalyst6500switchwithclassicalPort‐channels. Nexus7000connectedtoCatalyst6500switchesusingorphanports.
4.10.3 Configuration Details
Thefollowingconfigurationsareappliedtothetestnetwork:
Commonsystemcontrol,managementandaccounting:CommonsystemfeatureslikeSSH,Syslog,SNMP,NTPandManagementVRFareconfigured.
vPC:vPCtechnologyisdeployedinthenetworkbetweentheN7kandtheCatalystVSSswitchesasshowninthefigure3.
VLANtrunking:VLANtrunkingisusedintheaggregation‐accessblockstomaintainsegregationandsecurity.
STP:RapidSpanningTreeProtocolisusedtopreventLayer2loopsintheaggregation‐accessblocks.Thespanningtreerootisplacedontheaggregationlevel.RootGuardisconfiguredontheaggregationleveltoenforcerootplacement.BPDUFilter,BPDUGuardandPortFastEdgeareconfiguredontheaccessportstowardshosts.
LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork PVLAN:PVLANisconfiguredinthenetworkandisthemainfocusoftesting.The
followingPVLANcomponentsarecoveredinthenetwork: PVLANprimaryandsecondaryvlan(CommunityandIsolated) Promiscuoustrunkandsecondarytrunkonvpc PrivateVLANhostonclassicport‐channel. PrivateVLANpromiscuoustrunk,secondarytrunkonclassicPort‐channel
42
4.11 M1 vPC Scale Topology
4.12 DC1, ENT1 and M1 vPC Feature / Scale Coverage
FeatureMax DC1 ENT M1vPCF1 2 M1 5VDC 4FEX 6FEX‐HIF 96 ACL/ACE 500 500 POMembers 2VLAN/SVI 100 2000VLANperFEX 10FPVLAN 50MAC 7K 1K vPC 120VLANpervPC 1,10,16HSRPv2 100 2000OSPFNeighbor 16BGPNeighbor 2 18 PIMNeighbor 256 2000 PBRSequence 20 200
43
UnicastRoute 10K 22KMulticastRoute 500 3K MulticastSource 20MulticastOIF 10 MulticastGroup 100 390FabricExtender 3FPIS‐ISadjacencies 6 FPNumberofSwitchID’s 20PrimaryVLAN 20 140SecondaryVLAN 25 280Phy.PortsusedforPVLAN 468Port‐channel 125 388Port‐channelinPVLAN 18 388VPCPOwithPVLAN 10 256Hostmode 18 32PromiscuousAccess 2 64PromiscuousTrunk 2 186TrunkSecondary 7 186PVLAN 50 60vPCConfig‐Sync 120Netflow 84 WCCP 58
4.13 ISSU Matrix for DC1, ENT and M1 vPC Image ISSU/COLDBOOT DC1 ENT M1vPC7.2.0>7.2.0.upg ISSU PASS PASS PASS6.2.12>7.2.0 ISSU PASS PASS6.2.10>7.2.0 ISSU PASS6.2.8a>7.2.0 ISSU PASS 6.2.8b>7.2.0 ISSU PASS7.2.0>7.2.0.upg COLDBOOT PASS 6.2.12>7.2.0 COLDBOOT PASS6.2.10>7.2.0 COLDBOOT PASS 6.2.8a>7.2.0 COLDBOOT PASS 6.2.8b>7.2.0 COLDBOOT PASS PASS PASS7.2.0>6.2.12 COLDBOOT PASS PASS 7.2.0>6.2.10 COLDBOOT PASS7.2.0>6.2.8a COLDBOOT PASS7.2.0>6.2.8b COLDBOOT PASS PASS
44
5 NVT Findings/Conclusion/Recommendations Assigned/NewStillworkingonfixesandmaybeseeninCCOimageUnreproducibleNotseeninCCOimagemayhavefixedbyothercodefixes.Verified/ResolvedFixedinCCOimageClosedSystemlimitationandbehaviorwillremainthesameCSCus00094Symptom:IPFIBprocessmightcrashuponenablingLDPautoconfigfeature.Conditions: whilerunningthesystemimage6.2(10)andinpresenceofF3moduleswithMPLSfeaturesetenabled(configurationnotsupportedcurrently),theIPFIBprocessmightcrashuponenablingLDPautoconfig. Workaround:None,deploy7.2(0)instead.Severity:SevereStatus:ClosedPlatformSeen:Nexus7700ResolvedReleases:ApplicableReleases:6.2(10)CSCus00165Symptom: InaN7700systemrunning6.2(10),F3modulesmightfailtocomeonlinedueto“SAC_INTR:XbarASICinterruptoccurred:SACChicoInterruptDetected”Conditions: UnderthesameconditionsofCSCus00094, upontheIPFIBprocesscrash,alltheF3modulesmightfailtocomeonline. Workaround:None,deploy7.2(0)instead.Severity:SevereStatus:ClosedPlatformSeen:Nexus7700ResolvedReleases:ApplicableReleases:6.2(10)CSCus02115Symptom:M1failureuponupgradingthesystemimagefrom6.2(10)toa7.2(0)dailyimage.Conditions: whileupgradingthesystemimagefrom6.2(10)toaninterim7.2(0)build,oneoftheM1modulesfailedthebootupsequence. Workaround:reloadfailedmodule.Severity:SevereStatus:UnreproduciblePlatformSeen:Nexus7000ResolvedReleases:ApplicableReleases:7.2(0)CSCus22635Symptom:Sometime,theconfigurationoftheIPv6VIPmightnotsucceed.Conditions:Theproblemhasoccurredonlyonesanditcouldnotberecreatedanymore.Workaround:reloadthesystemSeverity:SevereStatus:UnreproduciblePlatformSeen:Nexus7000ResolvedReleases:ApplicableReleases:7.2(0)
45
CSCus31113Symptom:Sometimes,inascaledenvironment,MP‐BGProutesmightfailtobeproperlyinstalledintheRIBforoneormoreVRFs.Conditions:Sometimes,uponsystemreloadinascaledenvironment,someroutesorsomemroutesmightnotgetproperlyinstalledinthesharedmemory.Workaround: changethesettingforthe“limit‐resourcesu4route‐memminimumNNNmaximumNNN”andthenperformasystemreloadorasystemswitchoverinordertoensurethenewvalueisproperlyparsed.Toestimatethepropervalueforthememoryallocation,thefollowingcommandcanbeused“showrouting<ip|ipv4|ipv6|multicast>estimatememoryestimateroute<#ofroutes|mroutes>” Severity:MinorStatus:AssignedPlatformSeen:Nexus7000ResolvedReleases:ApplicableReleases:CSCus50951Symptom:Sometimes,inascaledenvironment,MP‐BGProutesmightfailtobeproperlyinstalledintheRIBforoneormoreVRFs.Conditions:Sometimes,uponsystemreloadinascaledenvironmentwithmanyhostroutes,someroutes(orsomemroutes)mightnotgetproperlyinstalledinthesharedmemory.Workaround: RestartBGPprocessorreloadthesystem
Severity:SevereStatus:UnreproduciblePlatformSeen:Nexus7700ResolvedReleases:ApplicableReleases:CSCut60771Symptom:InaMVPNsite,multicasttrafficsourcedlocallymightgetcontinuouslyduplicatedforthereceiversattachedtotheprimaryvPCorphanport.Conditions:InagivenMVPNsitewithbothlocalsourcesandreceivers,multicasttrafficmightgetcontinuouslyduplicatedforallthelocalgroupsforthereceiversattachedtothevPCprimaryorphanport.Workaround: None Severity:SevereStatus:Unreproducible PlatformSeen:Nexus7000ResolvedReleases:ApplicableReleases:7.2(0)CSCuu02810:Symptom:vlan_mgrtimedoutduringsystembringupwithcopyrsandreloadConditions:Aftercopyr&s,reloadofsystem,vdc,resultsinvlan_mgrtimeouterrorsWorkaround:noneSeverity:SevereStatus:ResolvedPlatform:Nexus7000ResolvedReleases:7.2.0ApplicableReleases:7.2.0CSCuu27998:
46
Symptom:("InvalidVLANID")communicatingwithMTS_SAP_VLAN_MGRConditions:Invalidnativevlanallocationduringavdcreload.Workaround:noneSeverity:ModerateStatus:AssignedPlatform:Nexus7000ResolvedReleases:ApplicableReleases:7.2.0CSCuu38691:Symptom:ISSUfaileddueto/nxos/tmpspaceexhaustionwithloggingConditions:ISSUfailsbeforepre‐checkandabortswithoutdoingthepre‐checkWorkaround:noneSeverity:ModerateStatus:AssignedPlatform:Nexus7000ResolvedReleases:ApplicableReleases:7.2.0CSCus21952Symptom:PVLANtrafficblock‐holeduringCOLDBOOTfrom7.2.0.512.binto6.2.10.binCondition: L3egresstrafficfromTrunksecondaryportwasnotworkingduetodummybdconfigurationforprimaryvlan.Workaround:Use6.2.12insteadof6.2.10Severity:Status:Resolvedandverifiedin6.2.12(fixnotavailablein6.2.10)Platform:Nexus7000ResolvedRelease:6.2.12ApplicableRelease:CSCuq47581Symptom:VPCPeer‐linkflapuponISSUto7.2.0Condition: ObjecttrackingfeatureinplaceWorkaround:RemoveobjecttrackingwhileperformISSUandadditbackSeverity:Status:CLOSEDPlatform:Nexus7000ResolvedRelease:ApplicableRelease:7.2.0CSCus71454Symptom:VPCprimarylegflapupononVPCpeerlinkflapCondition: VPC leg configured as Private VLAN host modeWorkaround:NoSeverity:2Status:APlatform:Nexus7000ResolvedRelease:ApplicableRelease:7.2.0
47
Symptom:FEXHIFconfigurationmisswhileCOLDBOOTfrom7.2.0to6.2.xCondition: COLD BOOT from 7.2.0 to 6.2.xWorkaround:Re‐applyHIFconfigurationafterFEXcomeonlineSeverity:Status:Platform:Nexus7000ResolvedRelease:ApplicableRelease:CSCuu39696Symptom:Removingpvlanhost‐associationfromoneVPCPOflapsallVPCPOCondition: VPC Po configured as community hostWorkaround:NoSeverity:2Status:APlatform:Nexus7000ResolvedRelease:ApplicableRelease:6.2.12&7.2.0
6 Appendix
6.1 VxLAN Solution Configuration Guide Thefollowingconfigurationsaredividedasbelow
‐ Spine‐ Leaf‐ LISPMapserver‐ LISPBranchrouter
Spine configuration guide N77K
o Featuresetrequiremento Installfeature‐setfabric#Installedfeature‐setfabrico Feature‐setfabric#Enable/Disablefabrico featurebgp#Enable/DisableBorderGatewayProtocolo featurelisp#Enable/DisableLocator/IDSeparationProtocolo featurepim#Enable/DisableProtocolIndependentMulticasto featureospf#Enable/DisableOpenShortestPathFirstProtocolo featureinterface‐vlan#Enable/Disableinterfacevlano featurelacp#Enable/DisableLACPo featurebfd#Enable/DisableBFDo featurenvoverlay#Enable/DisableNVOverlayo featurevni #Enable/DisableVirtualNetworkSegment(VNI)
o Globalconfigurationcommando nvoverlayevpn#Enable/DisableEthernetVPN(EVPN)o systembridge‐domain#ConfigureBridge‐domainIDo vni<range> #configureVNIrangeo ippimrp‐address<>#ConfigurestaticRPforgrouprangeforBUM
Traffico bridge‐domain<>#configureBridge‐domainID
o VRFConfiguration
48
o vrfcontext<>#CreateVRFandenterVRFmodeo Vni<> #configureL3VNIforVXLANo rdauto #GenerateRDautomaticallyo address‐familyipv4unicast#ConfigureIPv4addressfamilyo route‐targetimport<> #ImportTarget‐VPNcommunityo route‐targetimport<>evpn#SpecifyTargetforEVPNrouteso route‐targetexport<>#ExportTarget‐VPNcommunityo route‐targetexport<>evpn#SpecifyTargetforEVPNrouteso route‐targetbothauto#ExportAndImportTarget‐VPNcommunityo route‐targetbothautoevpn#SpecifyTargetforEVPNroutes
o MapthebridgedomaintoL3VNIo bridge‐domain<> #Bridge‐domainIDo membervni<> #configureL3VNIassociatedwithVRF
o Bridge‐domaininterface(eachVRFmusthave1bridge‐domain)o interfacebdi<> #Bdiinterfaceo noshutdown#Enable/disableaninterfaceo vrfmember<> #ConfigureVRFparameterso ipforward#Enableipforwardingoninterface
o Port‐channelinterface(forallL3links)o interfaceport‐channel1 #PortChannelinterfaceo mtu9216 #Configuremtufortheporto bfdinterval<>min_rx<>multiplier<>#BFDintervalcommandso noipredirects #disableIPdirected‐broadcasto ipaddress<> #ConfigureIPaddressoninterfaceo ippimsparse‐mode #onfiguressparse‐modePIMoninterfaceo Interfaceethernet<> #EthernetIEEE802.3zo mtu9216 #Configuremtufortheporto channel‐group1modeactive #Configureportchannelparameterso noshutdown #Enable/disableaninterface
o NVEinterface
o interfacenve1 #NVEinterfaceo noshutdown #Enable/disableaninterfaceo source‐interface<> #NVESource‐Interfaceo host‐reachabilityprotocolbgp #Configurehostreachabilityadvertisemento membervni<>associate‐vrf #Associatevniwithavrf
o BGPGlobalconfigurationo routerbgp1 #BorderGatewayProtocol(BGP)o address‐familyipv4unicast #Configureunicastaddress‐familyo network<loopback> #ConfigureanIPprefixtoadvertiseo redistributedirectroute‐map<passall> #Directlyconnectedo maximum‐paths<> #Numberofparallelpaths(EBGP)o maximum‐pathsibgp<> #Numberofparallelpaths(IBGP)
49
o address‐familyl2vpnevpn #ConfigureL2VPNEVPNaddress‐familyo route‐map<passall>permit10#Permitanyo vrf<> #ProductionVRFo address‐familyipv4unicast#Configureunicastaddress‐familyo network0.0.0.0/0 #Defaultrouteo advertisel2vpnevpn #advertisel2vpnevpnrouteso redistributedirectroute‐mappassall#Directlyconnected
o IBGP(SpineandLeafconfiguration)–SuggestiontoconfigureIPv4overinterfaceand
EVPNpeeringoverloopback
o templatepeerIBGP‐IPv4 #Templateconfigurationforpeerparameterso bfd #BidirectionalFastDetectionfortheneighboro remote‐as1 #SpecifyASNumberoftheneighboro password3<> #Configureapasswordforneighboro address‐familyipv4unicast #Configureunicastaddress‐familyo send‐community #SendCommunityattributetothisneighboro route‐reflector‐client #ConfigureaneighborasRoutereflectorcliento route‐mapSELFout #Applyroute‐maptoneighboro route‐mapSELFpermit10 #route‐mapnameo setipnext‐hoppeer‐address #Usepeeraddress(forBGPonly)o neighbor<>remote‐as1 #neighborIBGPpeeringo inheritpeerIBGP‐IPv4 #Inheritatemplateo update‐source<> #SpecifysourceofBGPsessionandupdates
o templatepeerIBGP‐EVPN #Templateconfigurationforpeerparameterso bfd #BidirectionalFastDetectionfortheneighboro remote‐as1 #SpecifyASNumberoftheneighboro password3<> #Configureapasswordforneighbor o update‐sourceloopback0 #SpecifysourceofBGPsessionandupdateso address‐familyl2vpnevpn #ConfigureL2VPNEVPNaddress‐familyo send‐communityextended #SendStandardandExtendedCommunity
attributeso route‐reflector‐client #ConfigureaneighborasRoutereflectorcliento neighbor<>remote‐as1 #neighborIBGPpeeringo inheritpeerIBGP‐EVPN #Inheritatemplate
o EBGP(Inter‐PODSpine)–SuggestiontoconfigureIPv4overinterfaceandEVPNpeeringoverloopback
o templatepeerEBGP‐IPv4 #Templateconfigurationforpeerparameterso bfd #BidirectionalFastDetectionfortheneighbor
o remote‐as10 #SpecifyASNumberoftheneighboro password3<> #Configureapasswordforneighbor o address‐familyipv4unicast #Configureunicastaddress‐familyo send‐communityextended #SendCommunityattributetothisneighboro neighbor<>remote‐as10 #neighborEBGPpeering
50
o inheritpeerEBGP‐IPv4 #Inheritatemplate
o templatepeerEBGP‐EVPN #Templateconfigurationforpeerparameterso bfd #BidirectionalFastDetectionfortheneighboro remote‐as10 #SpecifyASNumberoftheneighboro password3<> #Configureapasswordforneighboro update‐sourceloopback0 #SpecifysourceofBGPsessionandupdateso ebgp‐multihop2 #SpecifymultihopTTLforremotepeero address‐familyl2vpnevpn #ConfigureL2VPNEVPNaddress‐familyo send‐communityextended #SendStandardandExtendedCommunity
attributeso route‐mapUNCHGDout #Applyroute‐maptoneighboro route‐mapUNCHGDpermit10#route‐mapnameo setipnext‐hopunchanged #Useunchangedaddress(foreBGPsessiononly)o neighbor<>remote‐as10 #neighborEBGPpeeringo inheritpeerEBGP‐EVPN #Inheritatemplate
o IBGP (Spine and Legacy TOR configuration) – Legacy TORs eBGP peer‐to‐spine through "fake iBGP" by manipulating "local AS" in as‐path. So BGP peering is iBGP, but the local AS are
different, so the route gets redistributed to MP‐iBGP L2VPN EVPN.
o Routerbgp1 #BorderGatewayProtocol(BGP)o VRF<> #ProductionVRFo neighbor<>remote‐as201 #neighborEBGPpeeringo local‐as202no‐prependreplace‐as #Specifythelocal‐asnumberforthe
eBGPneighbor,Donotprependthelocal‐asnumbertoupdatesfromtheeBGPneighbor,Prependonlythelocal‐asnumbertoupdatestoeBGPneighbor
o address‐familyipv4unicast #Configureunicastaddress‐family
o IBGP(Intra‐PODSpineoverRinglinks)–OSPFoverringlinksanIPv4andEVPNoverloopbacko routerospf1 #OpenShortestPathFirst(OSPF)o router‐id<loopback> #SetOSPFprocessrouter‐id
o interfaceEthernet1/22o mtu9216 #Configuremtufortheporto bfdinterval<>min_rx<>multiplier3#BFDintervalo nobfdecho #disableBFDEchoo noipredirects #disableIPdirected‐broadcasto ipaddress<> #ConfigureIPaddressoninterfaceo iprouterospf1area0.0.0.0 #enableospfontheinterfaceo noshutdown #Enable/disableaninterface
51
o neighbor<>remote‐as1 #neighborIBGPpeeringo update‐sourceloopback0 #SpecifysourceofBGPsessionandupdateso address‐familyipv4unicast #Configureunicastaddress‐familyo send‐communityextended #SendStandardandExtendedCommunity
attributeso route‐mapLOCAL_PREFin #Applyroute‐maptoneighboro address‐familyl2vpnevpn #ConfigureL2VPNEVPNaddress‐familyo send‐communityextended #SendStandardandExtendedCommunity
attributeso route‐mapLOCAL_PREFin #Applyroute‐maptoneighbor
o route‐mapLOCAL_PREFpermit10#route‐mapnameo setlocal‐preference50 #BGPlocalpreferencepathattribute(configure
lessthandefault100)
o EBGP(corelink)–setuproute‐mapwhichwilladvertiseonlynon‐mobilitysubnets
o Routerbgp1 #BorderGatewayProtocol(BGP)o VRF<> #ProductionVRFo neighbor<>remote‐as11 #neighborIBGPpeeringo address‐familyipv4unicast #Configureunicastaddress‐familyo route‐mapWAN‐mapout #Applyroute‐maptoneighbor
o route‐mapWAN‐mappermit10 #route‐mapnameo matchipaddressprefix‐listwan‐list #Matchentriesofprefix‐lists
o ipprefix‐listwan‐listseq5permit<subnet>#onlyallownon‐mobilityprefixes
towardcoreandLISPbranch
LISP Configuration on Spine (N7K)
o vrfcontext<> #ProductionVRF o iplispetr #ConfiguresLISPEgressTunnelRouter
(ETR)o lispinstance‐id1000 #ConfiguresInstance‐IDforglobaldata‐
mappingso iplispitrmap‐resolver<ipadd>#TointeractwithMap‐Resolver
(primary)o iplispitrmap‐resolver<ipadd> #TointeractwithMap‐Resolver
(secondary)o iplispetrmap‐server<ipadd>key<>#AuthenticationkeyusedwithMap‐
Server(Primary)o iplispetrmap‐server<ipadd>key<>#AuthenticationkeyusedwithMap‐
Server(secondary)
o lispdynamic‐eid101 #Configuredynamic‐EIDsforroamingo database‐mapping<Mobilitynetwork><corelinkaddress>priority1
weight
52
#configureEID‐prefixandlocator‐setfordynamic‐EIDo register‐route‐notificationstag1#Registermore‐specificroutesofthe
database‐mappingEID‐prefixtoMap‐Server,Include
onlyrouteswiththisBGPtag
LISP configuration on Map server ( CSR1kv)
o routerlisp #Locator/IDSeparationProtocolo ddtauthoritativeinstance‐id1000<Subnet>#ddtauthoritativetoallow
(VXLANMobilitysubnet
o map‐server‐peer<ipaddress>#IPv4Peermap‐serverlocatoraddress(secondary)
o map‐server‐peer<ipaddress>#IPv4Peermap‐serverlocatoraddress(Primary)
o ddt #Enableddto site<name> #LISPsitenameo authentication‐key<key>#passwordo eid‐prefixinstance‐id1000<subnet>accept‐more‐specifics#allowVXLAN
Mobilitysubnet
o eid‐prefixinstance‐id1000<subnet>accept‐more‐specifics#allowBranchsubnet
o ipv4map‐server#ConfiguresaLISPMapServero ipv4map‐resolver#ConfiguresaLISPMapResolver(MR)
LISP configuration on Branch ( ASR1K) o routerlisp #Locator/IDSeparationProtocolo eid‐tabledefaultinstance‐id1000#ConfiguresInstance‐IDforglobaldata‐
mappings
o database‐mapping<Mobilitynetwork><corelinkaddress>priority1weight#configureEID‐prefixandlocator‐setfordynamic‐EID
o ipv4itrmap‐resolver<ipaddress>#TointeractwithMap‐Resolver(primary)o ipv4itrmap‐resolver<ipaddress>#TointeractwithMap‐Resolver(secondary)o ipv4itr#ConfiguresaLISPIngressTunnelRouter(ITR)o ipv4etrmap‐server<ipaddress>key<key>#AuthenticationkeyusedwithMap‐
Server(Primary)o ipv4etrmap‐server<ipaddress>key<key>#AuthenticationkeyusedwithMap‐
Server(secondary)
53
o ipv4etr #ConfiguresaLISPEgressTunnelRouter(ETR)
o ipv4map‐cache‐limit<> #Addressfamilyspecificmapcacheconfiguration
Leaf configuration guide N9K o Featuresetrequirement
o featurebgp#Enable/DisableBorderGatewayProtocolo featurepim#Enable/DisableProtocolIndependentMulticasto featureinterface‐vlan#Enable/Disableinterfacevlano featurelacp#Enable/DisableLACPo featurebfd#Enable/DisableBFDo featurenvoverlay#Enable/DisableNVOverlayo featurevni #Enable/DisableVirtualNetworkSegment(VNI)o featurevn‐segment‐vlan‐based#Enable/DisableVLANbasedVNsegmento featurevpc #Enable/DisableVPC(VirtualPortChannel)
o Globalconfigurationcommando nvoverlayevpn#Enable/DisableEthernetVPN(EVPN)o ippimrp‐address<>#ConfigurestaticRPforgrouprangeforBUM
Traffico fabricforwardinganycast‐gateway‐mac<>#AnycastGatewayMACoftheSwitch
o Vlantovn‐segmentmappingconfigurationcommand(NoteitisrequireforeachL2VLANandL3VLANperVRF)o vlan<> #Vlancommandso vn‐segment<> #VNSegmentidoftheVLAN
o VRFConfigurationo vrfcontext<>#CreateVRFandenterVRFmodeo Vni<> #configureL3VNIforVXLANo rdauto #GenerateRDautomaticallyo address‐familyipv4unicast#ConfigureIPv4addressfamilyo route‐targetimport<> #ImportTarget‐VPNcommunityo route‐targetimport<>evpn#SpecifyTargetforEVPNrouteso route‐targetexport<>#ExportTarget‐VPNcommunityo route‐targetexport<>evpn#SpecifyTargetforEVPNrouteso route‐targetbothauto#ExportAndImportTarget‐VPNcommunityo route‐targetbothautoevpn#SpecifyTargetforEVPNroutes
o MapthebridgedomaintoL3VNIo bridge‐domain<> #Bridge‐domainIDo membervni<> #configureL3VNIassociatedwithVRF
o VPCdomain<>
o VPNdomain #Specifydomaino peer‐switch #EnablepeerswitchonvPCpairswitches
54
o rolepriority<>#ConfigureprioritytobeusedduringvPCrole(primary/secondary)election
o system‐priority<>#Configuresystempriorityo peer‐keepalivedestination<>>source<>vrf<>#Keepalive/Hellowithpeer
switcho peer‐gateway #EnableL3forwardingforpacketsdestinedtopeer's
gatewaymac‐addresso ipv6ndsynchronize #DisplayNeighborDiscoveryinterfaceinformationo iparpsynchronize #CFSsynchronize
o VPCKeepalivelink
o interfaceport‐channel<> #PortChannelinterfaceo mtu9216 #Configuremtufortheporto vrfmemberVPC #Setinterface'sVRFmembershipo ipaddress<> #ConfigureIPaddressoninterface
o VPCpeer‐linko interfaceport‐channel<> #PortChannelinterfaceo switchportmodetrunk #Portmodetrunko switchporttrunkallowedvlan<>#SetallowedVLANcharacteristicswhen
interfaceintrunkingmode
o spanning‐treeporttypenetwork#Considertheinterfaceasinter‐switchlinko vpcpeer‐link #Specifyifthislinkisusedforpeer
communication
o VPVlegport‐channel(hostconnectedlink)
o interfaceport‐channel<> #PortChannelinterfaceo switchportmodeaccess #Portmodeaccesso switchportaccessvlan<> #Setaccessmodecharacteristicsoftheinterfaceo spanning‐treebpdufilterenable #EnableBPDUfilteringforthisinterfaceo mtu9216 #Configuremtufortheporto vpc<> #VirtualPortChannelconfiguration
o Vlaninterfaceconfiguration
o interfaceVlan101 #Vlaninterfaceo noshutdown #Enable/disableaninterfaceo mtu9216 #Configuremtufortheporto vrfmember<> #ConfigureVRFparameterso noipredirects #DisableSendICMPRedirectmessageso ipaddress<> #ConfigureIPaddressoninterfaceo noipv6redirects #DisablesendingICMPv6Redirect
messageso fabricforwardingmodeanycast‐gateway#AnycastGatewayForwardingMode
55
o VlaninterfaceconfigurationmappedtoL3VNI
o interfacevlan<> #Vlaninterfaceo noshutdown #Enable/disableaninterfaceo mtu9216 #Configuremtufortheporto vrfmember<> #ConfigureVRFparameterso noipredirects #ConfigureIPaddressoninterfaceo ipforward #Enableipforwardingoninterface
o Port‐channelinterface(forallL3links)
o interfaceport‐channel1 #PortChannelinterfaceo mtu9216 #Configuremtufortheporto bfdinterval<>min_rx<>multiplier<>#BFDintervalcommandso noipredirects #disableIPdirected‐broadcasto ipaddress<> #ConfigureIPaddressoninterfaceo ippimsparse‐mode #Configuressparse‐modePIMoninterfaceo Interfaceethernet<> #EthernetIEEE802.3zo mtu9216 #Configuremtufortheporto channel‐group1modeactive #Configureportchannelparameterso noshutdown #Enable/disableaninterface
o NVEinterface
o interfacenve1 #NVEinterfaceo noshutdown #Enable/disableaninterfaceo source‐interface<> #NVESource‐Interfaceo host‐reachabilityprotocolbgp #Configurehostreachabilityadvertisemento membervni<L3VNI>associate‐vrf #AssociateL3vniwithavrfo membervni<L2VNI> #NVEVN‐SegmentMembershipo mcast‐group<> #NVEMulticastGroup
o BGPGlobalconfigurationo routerbgp1 #BorderGatewayProtocol(BGP)o address‐familyipv4unicast #Configureunicastaddress‐familyo network<loopback> #ConfigureanIPprefixtoadvertiseo maximum‐pathsibgp<> #Numberofparallelpaths(IBGP)o address‐familyl2vpnevpn #ConfigureL2VPNEVPNaddress‐familyo route‐map<passall>permit10#Permitanyo vrf<> #ProductionVRFo address‐familyipv4unicast#Configureunicastaddress‐familyo advertisel2vpnevpn #advertisel2vpnevpnrouteso redistributedirectroute‐mappassall#Directlyconnected
o IBGP(SpineandLeafconfiguration)–SuggestiontoconfigureIPv4overinterfaceand
EVPNpeeringoverloopback
o templatepeerIBGP‐IPv4 #Templateconfigurationforpeerparameters
56
o bfd #BidirectionalFastDetectionfortheneighboro remote‐as1 #SpecifyASNumberoftheneighboro password3<> #Configureapasswordforneighboro address‐familyipv4unicast #Configureunicastaddress‐familyo send‐community #SendCommunityattributetothisneighboro o neighbor<>remote‐as1 #neighborIBGPpeeringo inheritpeerIBGP‐IPv4 #Inheritatemplateo update‐source<> #SpecifysourceofBGPsessionandupdates
o templatepeerIBGP‐EVPN #Templateconfigurationforpeerparameterso bfd #BidirectionalFastDetectionfortheneighboro remote‐as1 #SpecifyASNumberoftheneighboro password3<> #Configureapasswordforneighbor o update‐sourceloopback0 #SpecifysourceofBGPsessionandupdateso address‐familyl2vpnevpn #ConfigureL2VPNEVPNaddress‐familyo send‐communityextended #SendStandardandExtendedCommunity
attributeso neighbor<>remote‐as1 #neighborIBGPpeeringo inheritpeerIBGP‐EVPN #Inheritatemplate
o IBGPintervPCSVIlinkoverpeer‐link(OnlyforIPv4peering)andaddvlanidtopeerlinkallowlist
o interfaceVlan3901 #Vlaninterfaceo noshutdown #Enable/disableaninterfaceo bfdinterval<>min_rx<>multiplier<> #BFDintervalcommandso nobfdecho #DisableEchofunctionforalladdressfamilieso noipredirects #disableIPdirected‐broadcasto ipaddress<> #ConfigureIPaddressoninterfaceo noipv6redirects #DisablesendingICMPv6Redirectmessageso ippimsparse‐mode #Configuressparse‐modePIMoninterface
o routerbgp<> #BorderGatewayProtocol(BGP)o address‐familyipv4unicast #Configureunicastaddress‐familyo network<> #AddSVIinterfaceaddresso neighbor<>remote‐as1 #neighborIBGPpeeringo bfd #BidirectionalFastDetectionfortheneighboro password3<> #Configureapasswordforneighbor
o update‐source<sviid) #SpecifysourceofBGPsessionandupdateso address‐familyipv4unicast #Configureunicastaddress‐familyo send‐communityextended #SendStandardandExtendedCommunity
attributeso route‐reflector‐client #ConfigureaneighborasRoutereflectorcliento route‐mapNHSout #Applyroute‐maptoneighbor
o route‐mapNHSpermit10 #route‐mapnameo setlocal‐preference50 #BGPlocalpreferencepathattribute(configure
lessthandefault100)
57
o setipnext‐hoppeer‐address #Usepeeraddress(forBGPonly)
o EVPNConfigurationforeachL2VNI
o Evpn #EnterEVPNconfigurationmodeo vni<>l2 #ConfigureEthernetVPNIDo rdauto #VPNRouteDistinguisherAutoo route‐targetimportauto #ImportTarget‐VPNcommunityandGenerate
RTautomaticallyo route‐targetimport<> #RTextcommunityinaa:nnformato route‐targetexportauto #ExportTarget‐VPNcommunityandGenerate
RTautomaticallyo route‐targetexport<> #RTextcommunityinaa:nnformat
6.2 Service Provider MPLS Configuration Guideline Thefollowingconfigurationsareappliedtothetestnetwork:
Commonsystemcontrol,managementandaccounting:CommonsystemfeatureslikeSSH,TACACS+,Syslog,SNMP,NTP,SPAN,DNSandManagementVRFareconfigured.
o featuretacacs+ #enablingthetacacsfeatureo tacacsserverhost<ipaddress>key<0/7> #configurethetacacsserverto
authenticateuserso aaagroupservertacacs+<groupname> #enableservergroupsfor
redundancy server<ipaddress> use‐vrf<vrf_name> #use‐vrfbasedonserver
reachabilityo snmp‐serveruser<user‐name><group‐name>authmd5<pass‐phrase>priv
<pass‐phrase>localizedkey #snmpv3userwithauthenticationenabled
o ntpserver<ipaddress> #enablentpwithserveripaddresso ipdomain‐name<domainname> #enabledomain‐nameo interfacemgmt0 #configuremgmt0
vrfmembermanagement ipaddress<ip_address>
o powerredundancy‐modeps‐redundant #setthepowerinredundancymode
o nosystemadmin‐vdc #disablesystemadmin‐vdco installfeature‐setfabricpath #enablefeature‐setforfabricpatho installfeature‐setmpls #enablefeature‐setformplso vdc<VDC_name>id1 #spawnsanewnamedVDC
limit‐resourcemodule‐type<module‐type> #enable<module‐type>modulesforthegivenVDC
allowfeature‐setfabricpath #allowsfabricpathfeature‐settobeenabledonthegivenVDC
allowfeature‐setmpls #allowsmplsfeature‐settobeenabledonthegivenVDC
allocateinterface<interface‐ranges> #allocateoneofmoreinterfacerangestothegivenVDC.Whenassigninganyrangeofinterfacesthemaximumgranularityapplicableforthe“allocateinterface”commandislimitedtothelayoutoftheport‐groupforeachspecifictypeofmodule.Attachingtothemoduleandusing“showhardware
58
internaldev‐port‐map”allowstodisplaythemappingbetweenfrontpanelportsandASICinstances.
limit‐resourceu4route‐memminimum200maximum200 #allowstoallocatemoresharedmemoryfortheIPv4routes.ToestimatetheamountofmemorytobeallocatedforbothIPv4orIPv6routes,thefollowingtwocommandscanbeused:“showiproutesum”oneachVRFand“showrouting<ip|ipv4|ipv6>memoryestimateroutes<total‐number‐of‐routes>next‐hop<N>”.Likewise,thesamemethodcanbeappliedformulticastallocation.
o control‐plane service‐policyinputtest‐‐copp‐policy‐strict #enablesthetest—copp‐
policy‐strictforthecontrol‐planecoppcopyprofilestrictprefixmvpnclass‐maptypecontrol‐planematch‐anymvpn‐copp‐class‐normalmatchexceptionipmulticastrpf‐failurematchexceptionipv6multicastrpf‐failurecontrol‐plane #applythemodifiedmvpnclasstothecontrol‐planetoreducetheeffectsofPIMassertissue(CSCut68318)service‐policyinputmvpn‐copp‐policy‐strict
o snmp‐serveruseradminnetwork‐adminauthmd50x213ea412ed9e450340d412f4c9741a25 #enablesuseradminwithnetwork‐adminprivilegesandmd5authentication
o ippimauto‐rpforwardlisten #enablestoforwardauto‐rp
messages”o ipmsdporiginator‐id<interface> #enablestheoriginator‐idfor
MSDPmessageso ipmsdppeer<remote‐peer‐address>connect‐source<interface> #establishMSDP
peeringwithagivenremotepeer
o spanning‐treevlan<vlan‐ranges>priority<priority>#enablesspanningtreeforthespecifiedlistofvlanranges
o vrfcontext<vrf‐name> #createanewvrf ippimrp‐address<rp‐address>group‐list<mcast‐groups/mask> #
configurestaticRPforthespecifiedsetofmulticastgroupswithinthegivenvrf rd<N:M> #configuretheroute‐
discriminatorforthegivenvrf.WhendeployedinavPC/vPC+scenariositisrecommendedtousedifferentRDsonthetwovPCpeers.
mdtdefault<mcast‐group‐address> #configuretheMDTdefaulttunnelforthegivenvrf
mdtdata<mcast‐group‐address/mask>threshold<rate>#configureasetofMDTdatatunnelsforthegivenvrfdependingonthesizeoftheconfiguredmask
address‐familyipv4unicast #configureIPv4unicastAF
route‐targetimport<NN:M> route‐targetexport<NN:M>
59
o vpcdomain<N> #configurevPCdomainnumberN
peer‐switch #enablespeer‐switchonthetwovPCpeerstoactasasingledevicefortheSTP
peer‐keepalivedestination<remote‐peer>source<local‐peer>vrfkeepalive#configurethepeer‐keepaliveonthespecifiedvrf.
delayrestore180 #delaysthelegbringupfor3minutestoguaranteetheconvergenceoftheroutingprotocolstominimizeNStrafficdisruption
peer‐gateway #toallowthevPCpeerdevicetoactastheactivegatewayforpacketsaddressedtotheotherpeerdevicerouterMAC
fabricpathmulticastload‐balance #enablesload‐balancingofmulticasttrafficbetweenthetwovPC+peerdevices.
fabricpathswitch‐id1 #definestheemulatedswitch‐idforthefabricpathvPCpair
config‐sync #enableconfigsynchronizationbetweenthetwovPCpeers
iparpsynchronize #enableARPsynchronizationbetweenthetwovPCpeers
o SwitchedVirtualInterfaces(SVIs):
interfaceVlanN mtu9216 #enablejumboframes vrfmember<vrf‐name> #assigntheSVItoagivenvrf noipredirects #preventtheroutertosend
redirectsmessagestotheclients(ICMP) ipaddress<ip‐address>/<24bitmask> ipv6address<ipv4‐address>/<64bitmask> ipospfpassive‐interface #disabletheroutingupdateson
thespeficiedOSPFinterfacepreventingtheformationofanyOSPFadjacency iprouterospf1area0.0.0.0 #enableOSPFonthespecified
interfaceandplaceitinarea0 ippimsparse‐mode #enablePIMsparsemode hsrpversion2 #settheHSRPversiontwo hsrpG #definestheHSRPgroupGforthe
specifiedinterface preemptdelayminimum120 #delaysthepreemptionfortwo
minutesuponHSRPswitchovertominimizenetworkdisruptions priority101forwarding‐thresholdlower1upper101 #definethepriority
forthisHSRPpeer ip<hsrp‐ip‐address> #HSRPvirtualIPaddress hsrpGv6ipv6 #definestheHSRPv6groupGv6for
thespecifiedinterface preemptdelayminimum120 priority101forwarding‐thresholdlower1upper101 ip<hsrpv6‐ip‐address>
o LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork:
interfaceport‐channelN #createaport‐channelN mtu9216 #enablejumboframesontheport‐
channelN ipaddress<ip‐address>/<24bitmask>
60
ipospfnetworkpoint‐to‐point #definetheOSPFnetworktypeasp2ptominimizethetimetoformtheOSPFadjacency
iprouterospf1area0.0.0.0 #enableOSPFonthespecifiedinterfaceandplaceitinarea0
ippimsparse‐mode #enablePIMsparsemode interfaceport‐channelM #createthevPCpeer‐linkasa
switchedport‐channelcarryingallthenecessaryVLANs. switchport switchportmodetrunk switchporttrunkallowedvlan11‐510,2001‐2500 spanning‐treeporttypenetwork #definesthenetworktypeas
networkforthevPCpeer‐link mtu9216 #enablejumboframes vpcpeer‐link #definestheport‐channelMasthe
vPCpeer‐link interfaceport‐channelR switchport switchportmodetrunk switchporttrunkallowedvlan11‐60 mtu9216 #enablejumboframes vpcR #definestheport‐channelRasa
vPCleg
o MPLSLDPconfiguration: mplsldpconfiguration router‐idLo0force #definestheloopbackinterface0
asrouter‐idforalltheLDPupdates
o OSPF/LDPconfigurations:
routerospf1 #settheOSPFprocessidasone router‐id<router‐id> #settheOSPFrouter‐id mplsldpautoconfigarea0.0.0.0 #turnonLDPonalltheOSPF
interfacesconfiguredinthebackbonearea(refertosection5forfurtherinformation)
o BGP:iBGPisconfiguredbetweenthecoreswitchesandthetworoute‐reflectors.iBGPis
alsoconfiguredbetweeneachPEoneachsiteandthetwoRRs: routerbgp1 #definestheBGPASasone graceful‐restart‐helper #enablesthegraceful‐restart‐
helper address‐familyipv4unicast maximum‐pathsibgp2 #enableICMPfortheAFIPv4
unicast templatepeerRR‐CLIENT #definesthetemplatefortheRR
iBGPpeering remote‐as1 #iBGPsession update‐sourceloopback0 #usestheloopbackinterfacezero
asthesourceoftheBGPupdatesandmessages address‐familyipv4unicast route‐reflector‐client #setthetemplatetobeaclientfor
theroute‐reflectorclusterintheAFIPv4unicast
61
address‐familyvpnv4unicast send‐communityextended address‐familyvpnv6unicast send‐communityextended #allowstheextendedattributesfor
bothAFVPNv4andVPNv6 address‐familyipv4mdt #enablesMVPNservices neighbor<RR1‐loopback‐interface‐0> inheritpeerRR‐CLIENT neighbor<RR2‐loopback‐interface‐0> inheritpeerRR‐CLIENT #enableiBGPpeeringwiththetwo
route‐reflectorsconfiguredinacluster vrfvrf1 #definestheVPNv4parameters
suchasrouteinjections,routeredistributions,ECMP,etc.etc. address‐familyipv4unicast network<loopbackinterface>/<32bitmask> #injecttheloopback
interfacetobeadvertisedtotheotheriBGPpeers.ThisinterfaceisusedascustomerRPinterface
network<SVI1‐ip‐address>/<24bitmask> … network<SVIN‐ip‐address>/<24bitmask> #injecttheSVIsubnets
thatbelongtothespecifiedvrfintoiBGP maximum‐pathsibgp2 #enableECMP address‐familyipv4unicast
o IGMPv2:IGMPisusedbyhoststojoinmulticastgroupsofinterest.IGMPsnoopingis
enabledonallswitchesintheaggregation‐accessblockstopreventfloodingofmulticastdatatraffic. ipigmpsnooping #bydefaultenabledonNexus
o FP:FabricPathisdeployedintheaggregationblockDC1‐Dist‐N7k‐102.The
spinelayeriscomprisedofNexus7000switchesandtheleafswitchesaredeployedusingNexus5000switches:
feature‐setfabricpath #proceduretoinstallandenablefabricpathfeatureset
fabricpathtopology12 membervlan2001‐2500 #definesthesetofVLANsusedfor
thevPC+accessswitch(CE12) fabricpathtopology6 membervlan11‐260 fabricpathtopology7 membervlan261‐510 vlan11‐510,2001‐2500 modefabricpath #definesthe2fabricpath
topologiesdeployedforthe2CEdevices(CE6andCE7) fabricpathswitch‐id106 #definesthelocalFPswitch‐id vpcdomain100 #entersthevPC/vPC+parameters fabricpathmulticastload‐balance fabricpathswitch‐id1 #definestheemulatedFPswitch‐
id.ThisiscommonbetweenthetwovPCpeers interfaceport‐channel7 #ThisinterfaceisusedasvPC
peer‐link.IthastocarrybothFPandvPC+VLANsandithastobeconfiguredinFPmode
switchportmodefabricpath
62
fabricpathisismetric800 #sinceinsteadystateitisnotdesiredtoforwardtrafficthroughthevPCpeer‐link,itisrecommendedtosettheISISmetricthehighestintheFPdomain
fabricpathtopology‐member6 fabricpathtopology‐member7 fabricpathtopology‐member12 #FPtopologydefinitions interfaceport‐channel100 switchportmodefabricpath
fabricpathisismetric10 #inordertominimizetherootchangesformulticasttraffic,itisrecommendedtosettheISISmetricinalltheFPinterfaces.Thiswillminimizetheimpactonthetrafficconvergenceuponnetworkdisruptions
6.3 DC1 Configuration Guideline Thefollowingconfigurationsareappliedtothetestnetwork:
Commonsystemcontrol,managementandaccounting:CommonsystemfeatureslikeSSH,TACACS+,Syslog,SNMP,NTP,SPAN,DNSandManagementVRFareconfigured.
o featuretacacs+ #enablingthetacacsfeatureo tacacsserverhost<ipaddress>key<0/7> #configurethetacacsserverto
authenticateuserso aaagroupservertacacs+<groupname> #enableservergroupsfor
redundancy server<ipaddress> use‐vrf<vrf_name> #use‐vrfbasedonserver
reachabilityo snmp‐serveruser<user‐name><group‐name>authmd5<pass‐phrase>priv
<pass‐phrase>localizedkey #snmpv3userwithauthenticationenabled
o ntpserver<ipaddress> #enablentpwithserveripaddresso ipdomain‐name<domainname> #enabledomain‐nameo interfacemgmt0 #configuremgmt0
vrfmembermanagement ipaddress<ip_address>
BGP:eBGPisconfiguredbetweenthecoreswitchesandthepubliccloud.
o featurebgp #enablebgpo routerbgp<autonomous‐id> #bgpautonomous‐ido router‐id<router‐id>o graceful‐restartstalepath‐time<120>o log‐neighbor‐changeso address‐familyipv4unicasto redistributedirectroute‐map<acl‐name> #route‐mapusedforredistribution
directlyconnectedsubnetso redistributeospf1route‐map<acl‐name> #route‐mapusedforredistribution
OSPFrouteso maximum‐paths<8>o maximum‐pathsibgp<8>o neighbor<neighboripaddres>remote‐as100090 #BGPpeero address‐familyipv4unicasto prefix‐listNO_SELFin #aclconfiguredtorestrict
prefiximport
63
OSPF:OSPFistheIGPrunningacrossthenetwork.Eachaggregation‐accessblockisconfiguredasauniqueareawiththecoreswitchesplayingtheroleoftheABR.
o featureospf #enableospfforIPv4o featureospfv3 #enableospfforIPv6o routerospf<instance‐tag>o router‐id<ipaddress>o redistributebgp<as_no>route‐map<acl‐name> #route‐mapusedfor
redistributionforbgprouteso log‐adjacency‐changeso timersthrottlespf100200500o timersthrottlelsa50100300o auto‐costreference‐bandwidth1000000o default‐metric<1>
PIM‐SM:PIMSparseMode/PIMAnySourceMulticastisdeployedacrossthenetworkto
supportmulticast.Eachaggregation‐accessblockisconfiguredwiththeRPforthelocallysourcedgroups.
o featurepim #enablepimo ippimrp‐address<rp‐address>group‐list<multicast‐groups>#configurestatic
RPforamulticastgrouprangeo ippimsend‐rp‐announceloopback2prefix‐list<multicast‐groups>#configure
candidateauto‐rpo ippimsend‐rp‐discoveryloopback2 #configureauto‐rp
mapping‐agento ippimssmrange<> #configurepimssmfor
defaultrange o ippimauto‐rpforwardlisten #enableauto‐rpmessages
forwarding
MSDPAnycastRP:MSDPisdeployedtoexchangesourceinformationbetweenAnycastRPs.
o featuremsdp #enablemsdpo ipmsdporiginator‐id<interface> #configuresource
interfaceformsdppeering,generallyloopbackinterfaceo ipmsdppeer<ipaddress>connect‐source<interface> #configurepeer
address
vPC:ThevPCtechnologyisdeployedintheaggregation‐accessblockDC1‐Dist‐N7k‐101asshowninFigure1.Inaddition,dual‐sidedvPCisconfiguredbetweentheNexus7000andNexus5000switches
o featurevpc #enablevpco vpcdomain<domain‐id> #configurevpcdomain‐id
peer‐switch #enablepeer‐switchforfasterSTPconvergence
rolepriority200 #configurepriority peer‐keepalivedestination<ipaddress>source<ipaddress>vrfvpc‐
keepalive #configurekeep‐alivelink peer‐gateway #enablepeer‐gatewayto
avoidvPCloop track<id> #tracktheL3core
connectivitytoavoidblack‐hole iparpsynchronize #configurearp
synchronizeforfasterconvergenceofaddresstables
64
FP:FabricPathisdeployedintheaggregationblockDC1‐Dist‐N7k‐102.Thespinelayer
comprisesNexus7000switchesandtheleafswitchesaredeployedusingNexus5000switches.
o feature‐setfabricpath #configurefeature‐setfabricpath
o vlan<vlan‐range> modefabricpath #configurevlan‐rangein
fabricpatho fabricpathswitch‐id<switch‐id> #configureswitch‐ido vpcdomain<domain‐id> #configurevpcdomain‐id
fabricpathswitch‐id<vpc+_switch‐id> #configurevirtualswitchforpeerspresentonthenetwork
o interfaceport‐channel<po> #configurefabricpatinterface switchportmodefabricpath #configurefabricpath
STP:RapidSpanningTreeProtocolisusedtopreventLayer2loopsintheaggregation‐
accessblocks.Thespanningtreerootisplacedontheaggregationlevel.BPDUFilterandPortFastEdgeareconfiguredontheaccessportstowardsthehosts.
o interfaceport‐channel<po> #configureport‐channel switchport switchportaccessvlan<vlan> spanning‐treeporttypeedge #enablehost spanning‐treebpdufilterenable #configurebpdufilter
HSRP:HSRPisusedasthefirsthopgatewayprotocolforhosts.
o interfaceVlan<id> #configuresvio ipaccess‐group<acl>in #enableaccess‐listo ipaccess‐group<acl>outo noipredirectso ipaddress<ipaddress>o hsrpversion2o hsrp1
authenticationmd5key‐stringcisco #enableauthentication preemptdelayminimum200 priority200 ip<ipaddress> #HSRPIPaddress
FEX:FabricExtenders(Nexus2000)aredeployedonNexus7000
IGMP:IGMPisusedbyhoststojoinmulticastgroupsofinterest.IGMPsnoopingis
enabledonallswitchesintheaggregation‐accessblockstopreventfloodingofmulticastdatatraffic.
o ipigmpsnooping #bydefaultenabledonNexus
LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork.o featurelacp #enableLACP,bydefaultLACPis
usedonallport‐channel
UDLD:UDLDaggressivemodeisconfiguredacrossthenetworktodetectandpreventunidirectionallinks
o featureudld #enablefeatureudld
65
o udldaggressive #udldaggressivemodeisenabledtore‐establishtheconnectionwiththeneighbor
PVLAN:PrivateVLANconfiguredatDC101betweenNexus7000VPCpeersto: Nexus5000 CAT6500
FollowingPrivateVLANmodesconfigured: Promiscuous Isolated(host) Isolated(trunk) SinglePVLANassociationinaport‐channel(hostmode) MultiplePVLANassociationinaport‐channel(trunkmode) PVLANPromiscuousinhostmode PVLANPromiscuousintrunkmode
o featureprivate‐vlan #enablefeatureprivate‐vlano vlan<vlan‐id> #configureprimaryvlan
private‐vlanprimaryo vlan<vlan‐id>
private‐vlan<isolated/community> #configuresecondaryvlan private‐vlanassociation<vlan‐id> #configureassociationwith
primaryvlano interfaceport‐channel<port‐channel #configureport‐channel
switchport switchportmodeprivate‐vlantrunksecondary#PLANtrunkmode switchportprivate‐vlantrunkallowedvlan1 #configurenativevlan switchportprivate‐vlanassociationtrunk<primary><secondary>
o interfaceport‐channel<>
switchportswitchportmodeprivate‐vlanpromiscuous#PVLANPromiscuousswitchportprivate‐vlanmapping12011211‐1213#PVLANmappingvpc71#assignVPC
o interfaceport‐channel<>
switchportswitchportmodeprivate‐vlanhost#PVLANhostmodeswitchportprivate‐vlanhost‐association12011213#PVLANAssociationvpc81
6.4 ENT1 Configuration Guide Thefollowingconfigurationsareappliedtothetestnetwork:
Commonsystemcontrol,managementandaccounting:Commonsystemfeatureslike
SSH,TACACS+,Syslog,SNMP,NTP,SPAN,DNSandManagementVRFareconfigured.o featuretacacs+ #enablingthetacacsfeatureo tacacsserverhost<ipaddress>key<0/7> #configurethetacacsserverto
authenticateuserso aaagroupservertacacs+<groupname> #enableservergroupsfor
redundancy server<ipaddress>
66
use‐vrf<vrf_name> #use‐vrfbasedonserverreachability
o snmp‐serveruser<user‐name><group‐name>authmd5<pass‐phrase>priv<pass‐phrase>localizedkey #snmpv3userwithauthenticationenabled
o ntpserver<ipaddress> #enablentpwithserveripaddresso ipdomain‐name<domainname> #enabledomain‐nameo interfacemgmt0 #configuremgmt0
vrfmembermanagement ipaddress<ip_address>
BGP:eBGPisconfiguredbetweenthecoreswitchesandthepubliccloud.
o featurebgp #enablebgpo routerbgp<autonomous‐id> #bgpautonomous‐ido router‐id<router‐id>o graceful‐restartstalepath‐time<120>o log‐neighbor‐changeso address‐familyipv4unicasto redistributedirectroute‐map<acl‐name> #route‐mapusedforredistribution
directlyconnectedsubnetso redistributeospf1route‐map<acl‐name> #route‐mapusedforredistribution
OSPFrouteso maximum‐paths<8>o maximum‐pathsibgp<8>o neighbor<neighboripaddres>remote‐as100090 #BGPpeero address‐familyipv4unicasto prefix‐listNO_SELFin #aclconfiguredtorestrict
prefiximport
OSPF:OSPFistheIGPrunningacrossthenetwork.Eachaggregation‐accessblockisconfiguredasauniqueareawiththecoreswitchesplayingtheroleoftheABR.
o featureospf #enableospfforIPv4o featureospfv3 #enableospfforIPv6o routerospf<instance‐tag>o router‐id<ipaddress>o redistributebgp<as_no>route‐map<acl‐name> #route‐mapusedfor
redistributionforbgprouteso log‐adjacency‐changeso timersthrottlespf100200500o timersthrottlelsa50100300o auto‐costreference‐bandwidth1000000o default‐metric<1>
PIM‐SM:PIMSparseMode/PIMAnySourceMulticastisdeployedacrossthenetworkto
supportmulticast.Eachaggregation‐accessblockisconfiguredwiththeRPforthelocallysourcedgroups.
o featurepim #enablepimo ippimrp‐address<rp‐address>group‐list<multicast‐groups>#configurestatic
RPforamulticastgrouprangeo ippimsend‐rp‐announceloopback2prefix‐list<multicast‐groups>#configure
candidateauto‐rpo ippimsend‐rp‐discoveryloopback2 #configureauto‐rp
mapping‐agent
67
o ippimssmrange<> #configurepimssmfordefaultrange
o ippimauto‐rpforwardlisten #enableauto‐rpmessagesforwarding
MSDPAnycastRP:MSDPisdeployedtoexchangesourceinformationbetweenAnycastRPs.
o featuremsdp #enablemsdpo ipmsdporiginator‐id<interface> #configuresource
interfaceformsdppeering,generallyloopbackinterfaceo ipmsdppeer<ipaddress>connect‐source<interface> #configurepeer
address
vPC:vPCtechnologyisdeployedintheaggregation‐accessblockDC2‐Dist‐N7k‐201.Inaddition,dual‐sidedvPCisconfiguredbetweentheNexus7000andNexus5000switches.
o featurevpc #enablevpco vpcdomain<domain‐id> #configurevpcdomain‐id
peer‐switch #enablepeer‐switchforfasterSTPconvergence
rolepriority200 #configurepriority peer‐keepalivedestination<ipaddress>source<ipaddress>vrfvpc‐
keepalive #configurekeep‐alivelink
peer‐gateway #enablepeer‐gatewaytoavoidvPCloop
track<id> #tracktheL3coreconnectivitytoavoidblack‐hole
iparpsynchronize #configurearpsynchronizeforfasterconvergenceofaddresstables
STP:RapidSpanningTreeProtocolisusedtopreventLayer2loopsintheaggregation‐
accessblockDC‐Dist‐N7K‐201.MSTPisenabledonDC‐Dist‐N7K‐202forthesamepurposewhereverapplicable.Thespanningtreerootisplacedontheaggregationlevel.BPDUFilterandPortFastEdgeareconfiguredontheaccessportstowardshosts.
o interfaceport‐channel<po> #configureport‐channel switchport switchportaccessvlan<vlan> spanning‐treeporttypeedge #enablehost spanning‐treebpdufilterenable #configurebpdufilter
SNMP:SNMPtrapsareenabledandSNMPscriptsareusedtocollectsysteminformation
andtomonitorpotentialmemoryleaks. HSRP:HSRPisusedasthefirsthopgatewayprotocolforhosts.
o interfaceVlan<id> #configuresvio ipaccess‐group<acl>in #enableaccess‐listo ipaccess‐group<acl>outo noipredirectso ipaddress<ipaddress>o hsrpversion2o hsrp1
authenticationmd5key‐stringcisco #enableauthentication
68
preemptdelayminimum200 priority<priority> ip<ipaddress> #HSRPIPaddress
FEX:MultipletypesofFabricExtendersaredeployedonNexus5000parentswitches. IGMP:IGMPisusedbyhoststojoinmulticastgroupsofinterest.IGMPsnoopingis
enabledonallswitchesintheaggregation‐accessblockstopreventfloodingofmulticastdatatraffic.
o ipigmpsnooping #bydefaultenabledonNexus
LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork.o featurelacp #enableLACP,bydefaultLACPis
usedonallport‐channel
UDLD:UDLDaggressivemodeisconfiguredacrossthenetworktodetectandpreventunidirectionallinks
o featureudld #enablefeatureudldo udldaggressive #udldaggressivemodeisenabled
tore‐establishtheconnectionwiththeneighbor
RouteMAPforInter‐VRFPBR
o featurepbr #enablefeaturepbrroute‐mapGLOBAL‐to‐VRFpermit10 #defineroute‐map
matchipaddressPBR‐GLOBAL‐VRF#matchACLfortheroute‐map setvrfA#defineVRF
ipaccess‐listPBR‐VRF‐GLOBAL#defineACLfortheroute‐map 10permitip151.15.1.0/24any interfaceVlan1501#CreateSVIinterface vrfmemberA noipredirects ipaddress151.15.1.152/24 noipv6redirects iprouterospf1area0.0.0.151 ippimsparse‐mode ippimdr‐priority100 ippolicyroute‐mapVRF‐to‐global hsrpversion2 hsrp1501 authenticationtexteCATS priority100forwarding‐thresholdlower1upper100 timers13 ip151.15.1.1 ipdhcprelayaddress172.28.92.48 ipdhcprelayaddress172.28.92.49 noshutdown mtu9000
6.5 M1 vPC Scale Configuration Guideline Thefollowingconfigurationsareappliedtothetestnetwork:
69
Commonsystemcontrol,managementandaccounting:Commonsystemfeatureslike
SSH,Syslog,SNMP,NTPandManagementVRFareconfigured.o snmp‐serveruser<user‐name><group‐name>authmd5<pass‐phrase>priv
<pass‐phrase>localizedkey #snmpv3userwithauthenticationenabled
o ntpserver<ipaddress> #enablentpwithserveripaddresso ipdomain‐name<domainname> #enabledomain‐nameo interfacemgmt0 #configuremgmt0
vrfmembermanagement ipaddress<ip_address>
vPC:vPCtechnologyisdeployedinthenetworkbetweentheN7kandtheCatalystVSS
switchesasshowninthefigure3.
o featurevpc #enablevpco vpcdomain<domain‐id> #configurevpcdomain‐id
peer‐switch #enablepeer‐switchforfasterSTPconvergence
rolepriority200 #configurepriority peer‐keepalivedestination<ipaddress>source<ipaddress>vrfvpc‐
keepalive #configurekeep‐alivelink
peer‐gateway #enablepeer‐gatewaytoavoidvPCloop
iparpsynchronize #configurearpsynchronizeforfasterconvergenceofaddresstables
STP:RapidSpanningTreeProtocolisusedtopreventLayer2loopsintheaggregation‐
accessblocks.Thespanningtreerootisplacedontheaggregationlevel.RootGuardisconfiguredontheaggregationleveltoenforcerootplacement.BPDUFilter,BPDUGuardandPortFastEdgeareconfiguredontheaccessportstowardshosts.
o interfaceport‐channel<po> #configureport‐channel switchport switchportaccessvlan<vlan> spanning‐treeporttypeedge #enablehost spanning‐treebpdufilterenable #configurebpdufilter
LACP:LACPisusedforlinkaggregationtoformport‐channelsacrossthenetwork
o featurelacp #enableLACP,bydefaultLACPisusedonallport‐channel
PVLAN:PVLANisconfiguredinthenetworkandisthemainfocusoftesting.The
followingPVLANcomponentsarecoveredinthenetwork:o PVLANprimaryandsecondaryVLAN(Isolated)o SecondaryTrunkandpromiscuoustrunkonvPCo PrivateVLANpromiscuousaccessandPrivatevlanhostonclassisport‐channel.
70
o PrivateVLANpromiscuoustrunk,secondarytrunkonclassicPort‐channel
o featureprivate‐vlan #enablefeatureprivate‐vlano vlan<vlan‐id> #configureprimaryvlan
private‐vlanprimaryo vlan<vlan‐id>
private‐vlan<isolated/community> #configuresecondaryvlan private‐vlanassociation<vlan‐id> #configureassociationwith
primaryvlano interfaceport‐channel<port‐channel> #configureport‐channel
switchport switchportmodeprivate‐vlantrunksecondary switchportprivate‐vlantrunkallowedvlan1 #configurenativevlan switchportprivate‐vlanassociationtrunk<primary><secondary>
o interfaceport‐channel<port‐channel> #configureport‐channel switchport switchportmodeprivate‐vlantrunkpromiscuous switchportprivate‐vlanmappingtrunk<primarysecondary1,
secondary2,…> #configureprimarytosecondarymapping
o interfaceport‐channel<port‐channel> #configureport‐channel switchport switchportmodeprivate‐vlanpromiscuous #configurepromiscuous
access switchportprivate‐vlanmapping<primarysecondary>
#configureprimarytosecondarymappingo interfaceport‐channel<port‐channel> #configureport‐channel
switchport switchportmodeprivate‐vlanhost #configurehost
switchportprivate‐vlanhost‐association<primarysecondary> #configureprimarytosecondaryhost‐association