Enhancing IP/MPLS based Carrier Services to Address the Data Center Interconnect Market

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2024 Cisco Public

Enhancing IP/MPLS based Carrier Services to address the Data Center Interconnect Market Patrick Warichet: Technical Marketing Engineer NOSTG Vipul Shah: Engineering Product Manager NOSTG

•  Cisco TechAdvantage Webinars – May 1, 2013


Attending Cisco Live Orlando?

We are offering various Ethernet Services and Ethernet OAM learning opportunities at Cisco Live Orlando. Learn more and register at: http://www.ciscolive.com/us/attendees/education/

Technical Session Session Number Building Carrier Ethernet Services Using Cisco Ethernet Virtual Circuit Framework

BRKSPG-2204

Ethernet OAM: Technical Overview and Deployment Scenarios BRKNMS-2202

Redundancy Mechanisms for Carrier Ethernet and Layer 2 VPN Services

BRKSPG-2207

Towards Massively Scalable Ethernet: Technologies and Standards

BRKSPG-2206


Speaker & Panelist Introduction

Panelist Speakers

Patrick Warichet Technical Marketing

Engineer [email protected]

Tina Lam Engineering Product

Manager [email protected]

Vipul Shah Engineering Product

Manager [email protected]


Housekeeping

•  Submit questions in Q&A panel and send to !"##$%&'(#)*+*,$ Avoid CHAT window for better access to panelists

•  Please -./0#(+($+1($0.*+2(3('+$*453(6$

•  For 7(89:$&4;)., select COMMUNICATE > Join Audio Broadcast

•  Where can I get the presentation? Or send email to: &*<=+(-1&;3&'+&>(?-)*-.@-./$

•  A.)'$4*$B.5$.45$40-./)'>$C(-1";3&'+&>($7(8)'&5*D$$

EEE@-)*-.@-./F>.F+(-1&;3&'+&>($

$

•  For 7(89:$-&##$8&-<, click ALLOW phone button at the bottom of participants side panel


Agenda

•  Data Center Interconnect Requirements •  Data Center Interconnect Using Virtual Private LAN Service •  Data Center Interconnect Using Ethernet -VPN •  Summary

5


Data Center Interconnect Requirements

6


Data Center Interconnect (DCI) Concept

•  Data Centers are extending beyond traditional boundaries due to: !  Extending Operating System, File

System clusters, Database clusters !  Virtual/Physical machine mobility

due to load sharing, disaster prevention

!  Legacy devices/applications with embedded IP addressing

!  Time to deployment and operational reasons

!  Extend DC to solve power/heat/space limitations

Business Drivers

7

Business Drivers

IT Solutions

Disaster Prevention

Active/Standby Migration

Business Continuance

Server HA clusters, “Geo-clustering”

Workload Mobility

Move, consolidate servers, “Vmotion”

Connectivity flavor

Purpose Business Benefit

SAN extensions

Storing/replicating data

business continuity, disaster recovery, regulatory compliance

L3 extensions (L3VPNs)

Routed L3 Connectivity

Segmentation, Virtualization, file server backup applications

L2 extensions Layer 2 across data centers

High-Availability Clusters, Geo-clustering, Virtual machine mobility

Ethernet Emulation Service L2 L2

Main Data Center

Backup Data Center

Storage Storage

IP Routed Service

DWDM/CWDM

L3 L3

SAN SAN

L2 L2

FC FC


!"#$%&'()(&!*+)*,&

!"#$%&'()(&!*+)*,&

Workload increase

Core

Cloud VPN

!"#$%&'()(&!*+)*,&

Request resources

Workload Deployed

Additional Capacity Needs – Request Cloud Resources

1

Check Availability, Performance, Determine Optimal Location

2

Self-provision Network Tenant, Virtual Compute, Storage, VPN

3

vDC Active 4

1

2

3

4

A DCI use Case : Cloud Centric Networking

G%&62&*26.42>.H$B.5$-./04+(I$*+.5&>(I$'(+E.5<$

-+)*,+("&'()(&!*+)*,&

DCI

8


SP Managed DCI Solutions •  Standalone DCI network provides

interconnection between main SP DCs !  Owned by SP DC team

!  Addresses SP2SP only

!  Very high bandwidth – packet / optical solution likely the most cost effective

•  DCI Requires Technology Evolution in Data Center and SP NGN for: !  Multihoming

!  Scale (MAC-addresses, Number of Service Instances

!  Loadbalancing

!  Optimal Forwarding

!  Multicast optimization

!  Multitenancy

•  NGN Based DCI Interconnection models: !  Enterprise to Enterprise (E2E) !  Enterprise to Service Provider (E2SP) !  Service Provider to Service Provider

(SP2SP)

J%$KLM$ J%$KLN$

9'+$KLM$ 9'+$KLN$

J%$OPO$KL%9$

KL%9$

KL9$KL9$

%9$ %9$

L9$ L9$

9'+(505)*($KLQ$!8&-<$;..5,$

J+&';&#.'($KLQ$'(+E.5<$

9


SP Managed DCI

•  Data Centre Interconnection (DCI) refers to interconnection of data centres primarily in support of two use cases:

•  Workload mobility ! Moving workloads across Enterprise DCs ! Moving workloads from Ent to SP – “cloud bursting” ! Moving workloads from SP to SP – resilience and maintenance ! Needs mobility of the IP address – can be achieved at L2 and L3

•  Business Continuance - Server clustering and related transaction and storage replication ! Often requires L2 extension – but increasingly support for L3

Use Cases

10


VM Mobility •  Drivers for VM mobility

!  Maintenance without downtime

!  Workload balancing across sites !  Data centre migration, consolidation or

expansion !  Disaster avoidance

•  Example: VMware Vmotion !  Enables live migration of VMs !  Active memory and execution state of a VM is transmitted

from one physical server to another and access to the VMs disk storage is switched to the new physical host

!  VM retains its IP address and network connections, helping to ensure transparent migration

11

•  R(S4)5(/('+*$B.5$TU.V.'D$

U)'$WNNU80*$8&';E);+1$)*$5(S4)5(;$8(+E(('$;&+&$-('+5(*$&';$&+$#(&*+$MP80*$E)+1)'$;&+&$-('+5(*$

U&:$X/*$RCCI$)@(@$/&:$YZZ</$

[((0$U"L$&';$Q%$&;;5(**$4'-1&'>(;$&\(5$/.3($

•  J.45-($&';$;(*V'&V.'$*(53(5*$;.$'.+$'((;$+.$8($.'$*&/($*48'(+$)'$05&-V*(]$

^4+$+1(6$;.$'((;$+.$8($&8#($+.$/&)'+&)'$Q%$-.''(-V3)+6$

_N$(:+('*).'$)*$.'($E&6$.B$&-1)(3)'>$+1)*$


Local load balancing

Server Clustering

•  Global server load balancing (GSLB) distributes client traffic to servers across remote data centers !  e.g. using DNS, L3, route health injection, HTTP

redirection

•  GSLB commonly deployed with local Server Load Balancing (SLB) to Server Clusters

•  Server clustering technologies and storage sync / replication often require L2 extension – but increasingly support L3 also !  Active/Standby ; Heartbeat based !  Potentially out of span of control SP

Examples Microsoft MSCS*, Veritas Cluster Server*, Solaris Sun Cluster Enterprise, VMware Cluster, Oracle RAC (Real Appl.Cluster), HP MC/ServiceGuard, HP NonStop, HP Open VMS/TruCluster, IBM HACMP, EMS/Legato Automated Availability Mgr

Storage Array

Storage Array

Server Cluster

Server Cluster

Local load balancing

Global load balancing

K&+&$L('+5($`M$

K&+&$L('+5($`N$

12


SP Multitenant DCI: Baseline Requirements

•  Scales to the level required for SP virtual private cloud !  100s of thousands of MAC addresses per data

centre !  Thousands of tenants ; potentially more than 4K

service instances !  10s of data centres

•  Optimally forward unicast and multicast !  Shortest path !  Loop free !  Avoiding duplicates

•  Is resilient to all single element failures, i.e. in both NGN and DC

•  Provides control plane isolation between DCs

•  Fast to converge •  Uses network resources efficiently

!  All connections active with load balancing !  Flood minimisation

•  Easy to manage and operate •  Open standards based or clear track to

standardisation •  Integrates with SP NGN, whilst honouring any

administrative boundaries between DC and NGN, including DC connectivity across multiple AS’es

•  Supports geo-redundant PEs, i.e Enterprise DCI “back door”

•  Is DC transparent !  works for plain old spanning tree 802,1Q environment

(Normalized DCI Handoff) !  interworks with other DC technologies (Seamless DCI

Handoff)

13


SP Multitenant DCI Scaling

•  The scaling of a solution that uses L2 extension may be impacted by 4 concerns ! MAC address scalability

MAC address scalability is an issue on the DC WAN edge for solutions that use L2 extension Example: 250k CMAC addresses in a single SP data centre

This is multiplied across interconnected SP data centres and Enterprise data centres

! PW scale – (if PWs are used) ! Control plane scale – i.e. whether an L2 control plane domain is extended

between data centres ! Broadcast, Unknown unicast, and multicast traffic handling

14


Data Center Interconnect: Layer 2 Extension Technology

•  Virtual Private LAN Service (VPLS) ! The best available option in shipping code ! Does not meet some of the data center interconnect requirements for large SP

Multitenant Deployment options

•  Ethernet-VPN (E-VPN) / Provider Backbone Bridging Ethernet VPN (PBB-EVPN) ! New technologies to meet all of the large SP multitenant data center interconnect

requirements ! http://tools.ietf.org/html/draft-ietf-l2vpn-evpn ! http://tools.ietf.org/html/draft-ietf-l2vpn-pbb-evpn

15


Towards a common DCI Handoff ?

•  Is DCI a UNI or NNI ? ! All Service Instances

remapped to 802.1q VLANs ! or end to end (assumes

other encapsulation inside DC)

•  Is there a Control Plane inside the Data Center? ! Control Plane interworking

considerations !  IGP in DC ; BGP across

DCs ?

802.1Q/ad DC

802.1ah DC

Other IntraDC technologies : TRILL / 802.1Qaq/bp

DCI

16


Data Center Interconnect Using VPLS

17


./01&2&

!$3)#4*,&15$674*+)&

!1&

!1&

!1&

1)8*,+*)&.9-& 1)8*,+*)&.9-&

9/01&:&

9/01&;&9/01&<&

9/01&=&

0>&

./01&2&

!$3)#4*,&15$674*+)&

!1&

!1&

!1&

1)8*,+*)&.9-& 1)8*,+*)&.9-&

9/01&:&

9/01&;&9/01&<&

9/01&=&

0>&

Virtual Private LAN Services (VPLS)

•  Flooding (Broadcast, Multicast, Unknown Unicast)

•  Dynamic learning of MAC addresses on PHY and VCs

Virtual Forwarding Instance (VFI)

•  Forwarding Physical Port

Virtual Circuit

•  VPLS uses Split-Horizon and Full-Mesh of PWs for loop-avoidance in core

SP does not run STP in the core

•  RFC 4761 (BGP-Based VPLS); RFC 4762 (LDP-Based VPLS)

•  BGP Autodiscovery for LDP-Based VPLS RFC6074

•  Hierarchical VPLS option for LDP based VPLS !  802.1ad based

!  MPLS PW based

"00#)(*$$J0#)+2$a.5)b.'$

"00#)(*$$J0#)+2$a.5)b.'$

"00#)(*$$J0#)+2$a.5)b.'$

T%_J$K(c'(*$&'$"5-1)+(-+45($+.$%5.3);($L.''(-V3)+6$^(+E(('$P(.>5&01)-&##6$K)*0(5*(;$L4*+./(5$J)+(*$"-5.**$U"O*$&';$7"O*I$&*$QB$C1(6$7(5($L.''(-+(;$d*)'>$&$_"O@$

18


VPLS constraints

•  Not optimal with multicast ! Enhancements are maturing (using Label Switched Multicast with VPLS instead of

ingress resplication)

•  No active/active dual-homing per flow ! Per VLAN is possible

•  Does not hide customer mac-addresses •  PW scaling •  Handoff scaling and Service Instance Scaling ! 4k services per physical interface ! 000’s of VSI’s (hardware limitations)

19


Scaling VPLS : PBB-VPLS

•  VPLS current challenges ! MAC-Address Scalability at the PE ! Service Instance Scaling ! Limits DCI handoff to 4K services per interface

•  Approach: ! Use Provider Backbone Bridging (PBB)/802.1ah with

VPLS ! Hides Customer MAC-Addresses !  Described in

http://tools.ietf.org/html/draft-ietf-l2vpn-pbb-vpls-pe-model and http://tools.ietf.org/html/draft-ietf-l2vpn-pbb-vpls-interop

20


Multi-Homing DC’s and Loadbalancing/Resilience across VPLS

SP DC1 SP DC2

Ent DC1 Ent DC2

SP NGN

Signalling Active PE : Inter Chassis Control Protocol (ICCP,) or BGP Or PE Clustering (VSS, nV, VPC)

Multi Chassis Link Aggregation (MC-LAG) Multi Chassis Automatic Protection Switching (MC-APS)

Selecting correct Pseudo-Wire: Pseudowire Redundancy Or BGP

Flow Aware Transport: FAT-PW

21


Q%FU%_J$

Cisco A-VPLS : VPLS w/ MC-LAG & Fat-PWs

•  Flow Aware Transport (FAT) Pseudo-wires as in RFC6391 •  A: Aggregation switch performs EtherChannel flow-based hashing (on L2/L3/L4) & elects a link

towards VSS switch (e.g. Cat6000). •  B: VSS performs flow-based hashing (L2/L3/L4) to select outbound ECMP link. Optionally

inserts FAT-PW Flow Label (to be used in C). •  C: P nodes in MPLS core perform Loadbalaning

over ECMP using Flow Label. Note: Load-balancing decisions in A, B & C are independent.

Advanced 3-stage Load-Balancing

UL2_"P$

UL2_"P$

">>M$">>N$

UM$ UN$

UL2_"P$

UL2_"P$

"$ ^$ L$

TJJ$TJJ$


Summary Scaling VPLS

VPLS/LDP VPLS/BGP MAC-Address Scaling Needs PBB-VPLS Needs PBB-VPLS Flexible Service Instance Scaling

Needs PBB-VPLS Needs PBB-VPLS

Optimal Unicast (ECMP)

Needs Fat PWs Needs Fat PWs

Optimal Multicast Needs VPLS-LSM Needs VPLS-LSM Control Plane isolation between DC and DCI

None (Transparent to DC control plane except MST)

None (Transparent to DC control plane except MST)

Active Active per VLAN Needs ICCP and Pseudowire redundancy

BGP

Active Active per Flow Not achievable Not achievable Georedundancy Needs ICCP (active/

standby) BGP (active/standby)


DCI Using E-VPN / PBB-EVPN

24


Evolving Requirements for L2VPN

25

1.  All-active Redundancy •  Flow Based Load Balancing •  Flow Based Multi-pathing •  Geo-redundancy and Flexible Redundancy Grouping

2.  Simplified Provisioning and Operation •  Core Auto-Discovery •  Access Multi-homing Auto-Discovery •  New Service Interfaces

3.  Optimal Multicast with LSM •  P2MP Trees •  MP2MP Trees

4.  Fast Convergence •  Link/Port/Node Failure •  MAC Mobility

5.  Scalable for SP virtual private cloud service: •  Support O(10 Million) MAC Addresses per DC •  Confinement of C-MAC Learning

6.  Seamless interworking between TRILL / 802.1aq / 802.1Qbp and MST / RSTP •  Guarantee C-MAC Transparency on PE

7.  Fast Convergence •  Avoiding C-MAC Flushing

";;5(**(;$86$$

92T%O$e$%^^29T%O$Addressed by PBB-EVPN

Underline: Addressed by VPLS


What is Ethernet-VPN (E-VPN)

•  Treat MAC addresses as routable addresses and distribute them in BGP

•  When multiple PE nodes advertise same MAC, create multiple adjacencies in forwarding table

•  When forwarding traffic for a given unicast MAC DA, use hashing (L2/L3/L4) to pick one of the adjacencies

•  MP2MP or P2MP LSPs for Multicast Traffic Distribution

•  MP2P (like L3VPN) LSPs for Unicast Distribution

•  NO FULL MESH of PW’s !!!

At a glance

26

PE1/ MES1

PE4/ MES4

PE3 MES3

MPLS Core

PE2/ MES2

From PE1 iBGP L3-NLRI:

•  next-hop: n-PE1

•  <C-IP1, L1>

iBGP L2-NLRI •  next-hop: n-PE1

•  <C-MAC1, L2>

From PE3 iBGP L3-NLRI:

•  next-hop: n-PE3

•  <C-IP5, L1>

iBGP L2-NLRI •  next-hop: n-PE3

•  <C-MAC3, L2>


Ethernet Segment ID (ESI) and E-VPN Instance

•  MPLS Edge Switch (MES) is an E-VPN PE •  ESI Used when CE is multi-homed to two MES •  Allows remote MESes to load-balance to a multihomed CE •  10 octet value created by: ! Configuration ! Snooping of LACP (Multi-Homed Device) ! Snooping of STP BPDUs (Multi-Homed Network) !  (LLDP , although currently out of scope of the draft)

•  ESI has a new associated MPLS Label extended BGP Community (used for split-horizon procedures in multi-homing)

•  EVI = E-VPN Instance ID

27


EVPN Routes E-VPN Routes Purpose

Ethernet Auto-Discovery (AD) Route

Multi-purpose route for •  Auto-discovery of sites, vlans, VPNs, and site/vlan •  Load Balancing •  Split Horizon filtering on an Ethernet segment

MAC Route •  Advertisement of Client/Provider MACs

Inclusive Multicast Route •  Auto-discovery of MESes belonging to a VPN for setting up multicast tree

Selective Multicast Route •  Auto-discovery of MESes belonging to a given c-(S,G) of a VPN for setting up a multicast tree

Ethernet Segment Route •  DF election on an Ethernet segment •  MC-LAG state synch on an Ethernet segment •  Loop prevention on an ES during transient state

Leaf Auto-Discovery (AD) Route •  Multicast Tree Auto-discovery

28


E-VPN Operations Example

!  Host M1 sends a message with MAC SA = M1 and MAC DA=bcast

!  PE1 learns M1 over its Agg2-PE1 AC and distributes it via BGP to other PE devices

!  All other PE devices learn that M1 sits behind PE1

M1 communicates with M2 (e.g. ARP) - Broadcast

29

PE1

PE2 PE4

PE3

AGG1

AGG2

AGG3

AGG4

AGG5

AGG6

ESI=1

ESI=3

ESI=2

C-MAC1

C-MAC2

iBGP L2-NLRI •  next-hop: n-PE1 •  <C-MAC1, Label 100>

BGP

M1

M2


E-VPN Operations Example – Cont.

!  Host M2 sends response with MAC SA = M2 and MAC DA = M1

!  PE4 learns M2 over its Agg5-PE4 AC and distributes it via BGP to other PE devices

!  PE 4 forwards the frame to PE1 since it has learned previously that M1 sits behind PE1

!  All other PE devices learn that M2 sits behind PE4

Reply from M2 to M1 (Unicast)

PE1

PE2 PE4

PE3

AGG1

AGG2

AGG3

AGG4

AGG5

AGG6

ESI=1

ESI=3

ESI=2

M1

M2

iBGP L2-NLRI •  next-hop: n-PE1 •  <C-MAC2, Label 200>

100

30

BGP


It looks easy but not so fast ! •  In the shown example, how do we

ensure that !  ARP broadcast packet doesn’t get loopback to the

originating Agg device (Agg-1) : Split Horizon for ESI

!  Either PE3 or PE4 forward the broadcast frame to the far-end dual-homed device (Agg-2) Designated Forwarder Selection

!  When PE3 wants to forward a packet with destination address MAC1, it needs to send it to both PE1 and PE2 even though it only learned MAC1 from PE1 Aliasing

31

?@!=& ?@!<&

?@!=& ?@!<&

?@!=& ?@!<&

%9M$

%9N$

%9f$

%9Y$

%9M$

%9N$

%9f$

%9Y$

">>M$ ">>N$

">>M$ ">>N$

">>M$ ">>N$%9M$

%9N$

%9f$

%9Y$


Split Horizon for Ethernet Segment

•  PE1 advertises in BGP a split-horizon label associated with the ESI-1 (in the Ethernet AD route)

•  Split-horizon label is only used for multi-destination frames (unknown unicast, mcast, bcast)

•  When PE1 wants to forward a multi-destination frame, it appends this SH label to the packet

•  PE2 uses this label to perform split-horizon filtering for frames destined to ESI-1 ! A frame originated by a segment must not be

received by the same segment

32

MAC1 MAC2 PE1

PE2

PE3

PE4

Agg1 Agg2

ESI-1 ESI-2


Designated Forwarder (DF) Election

•  Only the PE that is the DF shall forward the multi-destination frame to an ESI

•  DF election is performed by each PE advertising the ESI in BGP (Ethernet AD route) for a given VLAN

•  PE with highest (or lowest) IP address is selected ad the DF for that segment

•  Non-DF PEs block that port for that segment – e.g., do not transmit any multi-destination frames to that segment

33

MAC1 MAC2 PE1

PE2

PE3

PE4

Agg1 Agg2

ESI-1 ESI-2


Aliasing

•  In order for PE3 to be able to perform load balancing among the flows destined to MAC1 on ESI-1, it needs to know that: a)  ESI-1 sits behind both PE1 and PE2 b)  MAC-1 is associated with ESI-1

•  PE1 and PE2 use Ethernet AD route to advertise ESI-1

•  PE1 uses MAC route to advertise MAC1 •  All the remote PEs (PE3 and PE4) use these

two routes in combination to associate a)  MAC1 to ESI-1 b)  subsequently MAC-1 to [PE1 and PE2]

34

MAC1 MAC2 PE1

PE2

PE3

PE4

Agg1 Agg2

ESI-1 ESI-2


Reducing ARP Flooding in E-VPN

•  PE/MESes perform Proxy ARP wthe destination is in the same subnet as the sender of the ARP request

•  The ARP request is not forwarded to other PE/MESes •  How does the PE/MES learn the IP address bound to the MAC address

when the MAC address is remote? ! A PE/MES advertises the local IP to MAC bindings in the BGP MAC routes

•  How does a PE/MES learn the IP to MAC binding when the MAC address is local? ! Control or management plane between MES and CEs or data plane snooping

(DHCP, etc)

35


Provider Backbone Bridging E-VPN (PBB-EVPN)

•  Advertise local B-MAC addresses in BGP to all other PEs that have at least one VPN in common just like E-VPN

•  Build a forwarding table from remote BGP advertisements just like E-VPN (e.g., association of B-MAC to MPLS labels)

•  PEs perform PBB functionality just like PBB-VPLS C-MAC learning for traffic received from ACs and C-MAC/B-MAC association for traffic received from core

36

PE2

PE1

PE3

LACP CE1

MPLS

BE B BE B

<- E-VPN |PBB ->

<- PBB|E-VPN ->

• $J)'>#($^2U"L$+.$

5(05(*('+$*)+($QK$

• $-&'$;(5)3($+1($^2U"L$

&4+./&V-&##6$B5./$

*6*+(/$U"L$&;;5(**$

.B$_"L%$

B-MAC = Site ID

TRILL

B-MAC BGP Routes


PBB-EVPN Main Principles

•  DF Election with VLAN Carving !  Prevent duplicate delivery of flooded frames. !  Uses BGP Ethernet Segment Route. !  Performed per Segment rather than per (VLAN,

Segment). !  Non-DF ports are blocked for flooded traffic (multicast,

broadcast, unknown unicast). •  Split Horizon for Ethernet Segment

!  Prevent looping of traffic originated from a multi-homed segment.

!  Performed based on B-MAC source address rather than ESI MPLS Label.

•  Aliasing !  PEs connected to the same multi-homed Ethernet

Segment advertise the same B-MAC address. !  Remote PEs use these MAC Route advertisements for

aliasing load-balancing traffic destined to C-MACs reachable via a given B-MAC.

37

%9$ %9$

%9$ %9$

%9$ %9$

%9$ %9$

%9$ %9$

%9$

^2U"LM$

^2U"LM$


Operation: Dual Homed Device

•  Each PE advertises a MAC route per Ethernet Segment (carries B-MAC associated with Ethernet Segment). !  Both PEs advertise the same B-MAC for the same Ethernet Segment.

•  Remote PE installs both next hops into FIB for associated B-MAC.

!  Hashing used to load-balance traffic among next hops.

•  PE1 MAC Routes:

!  Route: RD11, B-MAC1, RT2, RT3

•  PE2 MAC Routes:

!  Route: RD22, B-MAC1, RT2, RT3

38

PE1

PE2

VLAN 2, 3

VLAN 2,3

B-MAC1

PE3

VPN B-MAC NH

RT3 B-MAC1 PE1

RT3 B-MAC1 PE2

RT2 B-MAC1 PE1

RT2 B-MAC1 PE2

RIB

VPN B-MAC NH

RT3 B-MAC1 PE1, PE2

RT2 B-MAC1 PE1, PE2

FIB

MPLS/ IP


Operation: Dual Homed Network

•  Each PE advertises two MAC routes per Ethernet Segment: one with high Local Pref for active VLANs, and another with low Local Pref for standby VLANs.

•  Remote PE installs the route with higher local pref into FIB for associated B-MAC.

•  PE1 MAC Routes: !  Route 1: RD11, B-MAC1, RT3, Local Pref = High

!  Route 2: RD12, B-MAC1, RT2, Local Pref = Low

•  PE2 MAC Routes: !  Route 1: RD22, B-MAC2, RT3, Local Pref = Low

!  Route 2: RD21, B-MAC2, RT2, Local Pref = High

39

PE1

PE2

VLAN 3

VLAN 2

PE3 VPN B-MAC NH Pref

RT3 B-MAC1 PE1 High

RT3 B-MAC2 PE2 Low

RT2 B-MAC1 PE1 Low

RT2 B-MAC2 PE2 High

RIB

VPN B-MAC NH

RT3 B-MAC1 PE1

RT2 B-MAC2 PE2

FIB

MPLS/ IP Ethernet B-MAC1

B-MAC2


Advantages: MAC Address Scalability

1.  BGP MAC Advertisement Route Scalability !  Multiple orders of magnitude difference between C-MAC & B-MAC addresses

2.  C-MAC Address Confinement !  With data plane C-MAC learning, C-MACs are never in RIB and are only present in FIB for active flows !  Whereas, with control plane C-MAC learning, C-MACs are always in RIB and maybe also in FIB

40

7"O$

KL$J)+($M$

KL$J)+($N$

KL$J)+($O$

ghMUi$L2U"L*$

ghMZZi$^2U"L*$


Advantage: Avoiding C-MAC Address Flushing

•  B-MAC represents a site (Ethernet Segment)

•  Link, port, or node failure doesn’t change the B-MAC address to C-MAC address bindings – only changes the next hop for that B-MAC

41

PE1

PE2

VLAN 3

VLAN 2

B-MAC1

PE3 RIB

C-MAC VPN B-MAC NH

CM1 RT3 B-MAC1 PE1 PE2

CM2 RT2 B-MAC1 PE2

FIB

MPLS/ IP

VPN B-MAC NH Pref

RT3 B-MAC1 PE1 High

RT3 B-MAC1 PE2 Low

RT2 B-MAC1 PE1 Low

RT2 B-MAC1 PE2 High

1

2 Withdraw B-MAC1

3 Remove

4 Update


Advantage: IntraDC Interworking

•  End-to-end tunneling of C-MAC addresses thus avoiding data-plane termination and C-MAC learning by PE.

•  Control plane isolation between different TRILL / IEEE 802.1aq/ 802.1Qbp islands.

TRILL / IEEE 802.1aq / 802.1Qbp

42

PBB-EVPN TRILL /

802.1aq/ 802.1Qbp

IS-IS IS-IS BGP

TRILL/PBB Encapsulation

Control Plane

Data Plane

PE PE

MPLS

TRILL / 802.1aq/ 802.1Qbp


Additional Advantages

•  C-MAC Mobility with MAC sub-netting ! Typically C-MACs are not managed and thus can not be sub-netted ! B-MACs on the other hand are always managed and can easily be sub-netted ! Even when C-MACs are sub-netted, VM mobility contradicts the effect of sub-

netting

•  Avoid transient loop for known unicast when doing egress MAC lookup ! Since B-MAC SA is always transmitted with every frame, checking of every frame

against its source MAC SA for known unicast frame is already provided by PBB

43


BGP Routes Comparison

44

Route E-VPN PBB-EVPN

MAC Advertisement Route Distribute C-MAC • Distribute B-MAC • Aliasing • Fast Withdraw

Ethernet A-D Route • Aliasing • Split-Horizon Label • Fast Withdraw • DF Election

Not Used

Ethernet Segment Route DF Election DF Election

Inclusive Multicast Route Multicast Tree Discovery Multicast Tree Discovery

Selective Multicast Route Multicast Tree Discovery

Multicast Tree Discovery

Leaf Discovery Route Multicast Tree Discovery Multicast Tree Discovery


Comparison of Solutions

45

Requirement E-VPN PBB-EVPN

All-Active Redundancy with Flow Based Load-balancing ! !

Flow Based Multi-pathing ! !

Geo-redundancy and Flexible Redundancy Grouping ! !

Core Auto-Discovery ! !

Access Multi-homing Auto-Discovery ! !

New Service Interfaces ! !

LSM with P2MP Tree ! !

LSM with MP2MP Tree ! !

Fast Convergence on Failure ! !

Fast Convergence on MAC Mobility ! !

Fast Convergence: Avoiding C-MAC Flushing !

Scale to Millions of MAC Addresses !

Confinement of C-MAC Learning !

Seamless interworking between TRILL /802.1Qaq/802.1Qbp ! (C-MAC Transparency issue) !


VPLS and EVPN Scaling

•  Actual requirements from a large SP: !  n DCs = 20 !  d cmacs per DC = 500k !  2 PEs per DC !  c customers (with bridge domain per customer) = 8k !  l linecards per PE = 8 !  e edge switches per DC on average = 8

SP DCI Case Study

46

# Model Mac scale reqs per PE PW / LSP scale reqs per PE 1a VPLS 500k*20 = 10M 8k*(2*20-1) = 312k

1b VPLS + Cluster 500k*20 = 10M 8k*(20-1) = 152k

2 EVPN 500k*20 = 10M 2*20-1 = 39

3 PBB-VPLS (PE linecard is I component) 500k*20/8 = 1.25M 2*20-1 = 39

4 PBB-EVPN (PE linecard is I component) 500k*20/8 = 1.25M 2*20-1 = 39

5 EVPN + TRILL/FP 20*8 = 160 2*20-1 = 39


VPLS and EVPN Scaling

•  Parameters !  n DCs !  d cmacs per DC !  2 PEs per DC

!  c customers (with bridge domain per customer)

!  l linecards per PE !  e edge switches per DC

on average

Requirements

47

# Model Mac scale reqs per PE

PW / LSP scale reqs per PE

1a VPLS d*n c*(2n-1)

1b VPLS + cluster d*n c*(n-1)

2 EVPN d*n 2n-1

3a PBB-VPLS (PE is I-component) d*n 2n-1

3b PBB-VPLS (PE linecard is I-component)

d*n/l 2n-1

4a PBB-EVPN (PE is I-component) d*n 2n-1

4b PBB-EVPN (PE linecard is I-component)

d*n/l 2n-1

5 EVPN + TRILL/FP n*e 2n-1


Comparison DCI MPLS solutions

48

Characteristics Legacy VPLS Cisco’s A-VPLS E-VPN PBB-EVPN

Flow-based Load Balancing No Yes Yes Yes

Flow-based multi-pathing No Yes Yes Yes

Geo redundant group & opt. unicast No No Yes Yes

Flexible redundancy grouping No No Yes Yes

MAC Scaling No No No Yes

MP2MP MDT support No No Yes Yes

P2MP MDT support No No Yes Yes

Fast convergence upon AC failure No Yes Yes Yes

Flow-based or VLAN-based LB for MHN No Yes Yes Yes

Minimal configuration No Yes Yes Yes

Auto detect of MHN/MHD for flow-based LB No No Yes Yes

Scaling MPLS Core – full-mesh No No Yes Yes


Summary

• Data Center Interconnect Requirements

• Data Center Interconnect Using Virtual Private LAN Service

• Data Center Interconnect Using Ethernet -VPN

49


Key Take Aways

•  Data Center Interconnect is a new Service that can be offered by SP’s by enhancing their IP/MPLS Services !  L2 Ethernet Service !  L3 Service

•  VPLS with enhancement can be leveraged to offer L2 Ethernet Service •  E-VPN solves some of the VPLS Scaling issues ! Active, Active ! Better Multicast support ! Pseudo Wire Scaling ! Autoconfiguration

•  PBB-EVPN solves all of the Scaling issues associated with multi-tenant, high-capacity DCI ! Service Instance Scaling ! MAC Address Scaling !  Interworking with next gen IntraDC Technologies

50


Poll Question #1

How many Datacenters does your company have? A.  1-2 B.  2-3 C.  4-5 D.  6+

51


Poll Question #2

Which platform are you considering or currently have at the DCI edge? (select multiple choices, as applicable) A. ASR 9K B. Nexus 7K C. ASR 1K D. Catalyst 6K E. Another vendor equipment

52


Poll Question #3

Are you considering VPLS/PBB-EVPN/EVPN for your Data Center Interconnect technology? A.  I would consider deployment in the next 12 months B.  I would consider deployment in the next 12-24 months C. Undecided. Need further evaluation / understanding of the technology D. No plans to deploy / Not applicable to my environment E.  I am considering another DCI technology at this time

53


Poll Question #4

Which features are most critical for your company when considering a DCI technology? (select multiple choices, as applicable) A. Route scaling associated with Multi-tenant networks B. Seamless interworking between intra-DC and DCI protocols C. Resiliency and load balancing D. Multicast E. Other

54


Glossary

55

(G)SLB (Global) Server Load Balancing

(V)LAN (Virtual) Local Area Network AD Auto-Discovery ARP Address Resolution Protocol AS Autonomous System BGP Border Gateway Protocol B-MAC Backbone MAC BPDU Bridge Protocol Data Unit CE Customer Edge C-MAC Customer MAC CWDM Coarse Wave Division Multiplexing DC Data Center DCI Data Center Interconnect DF Designated Forwarder DHD Dual Homed Device DHN Dual Homed Network DWDM Dense Wave Division Multiplexing ECMP Equal Cost Multi Path ESI Ethernet Segment ID EVI Ethernet VPN Instance E-VPN Ethernet VPN FAT Flow Aware Transport FC Fiber Channel FIB Forwarding Information Base ICCP Inter Chassis Control Protocol IGP Interior Gateway Protocol IP Internet Protocol L2-VPN Layer 2 VPN LACP Link Aggregation Control Protocol LDP Label Distribution Protocol LLDP Link Layer Discovery Protocol

LSM Label Switched Multicast MAC Media Access MC-APS Multi-Chassis Automatic Protection Switching MC-LAG Multi-Chassis Link Aggregation MES MPLS Edge Switch MHN Multi Homed Network MP2MP Multipoint to Multipoint MP2P Multipoint to Point MPLS Multi Protocol Label Switching MST Multiple Spanning Tree Protocol NGN Next Generation Network NLRI Network Layer Reachability Information NNI Network to Network Interface nV Network Virtualization P2MP Point to Multipoint P2P Point to Point PBB-EVPN Provider backbone Bridging Ethernet VPN PE Provider Edge PW Pseudo Wire RD Route Distinguisher RIB Routing Information Base RSTP Rapid Spanning Tree Protocol RTT Round Trip Time SAN Storage Area Network TRILL Transparent INterconnection of a Lot of Links UNI User to Netwok Interface VPLS Virtual Private LAN Service VPN Virtual Private Network VSI Virtual Switch Instance VSS Virtual Switch WAN Wide Area Network


•  A8(+B&C#$D&$•  %#(&*($-./0#(+($+1($0.*+2(3('+$*453(6$

•  A.)'$4*$B.5$40-./)'>$E(8)'&5*D$

R(>)*+(5D$EEE@-)*-.@-./F>.F+(-1&;3&'+&>($

$

j.##.E$4*$$$$$$$?P(+k.45^4)#;g'$

$

Enhancing IP/MPLS based Carrier Services to Address the Data Center Interconnect Market

Technology

Transcript of Enhancing IP/MPLS based Carrier Services to Address the Data Center Interconnect Market