Evolving Data Center Switching - Amazon Web...
Transcript of Evolving Data Center Switching - Amazon Web...
![Page 1: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/1.jpg)
Evolving Data Center Switching
Part 1. Setting the Stage for Layer 2 Multi-pathing (TRILL)
Brad HedlundCisco Systems, Inc.
BRAD HEDLUND .COM
v 1.5
5/26/2010
![Page 2: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/2.jpg)
- About the Author -
Brad HedlundTechnical Solutions Architect, Data CenterCisco Systems, Inc.CCIE #5530
Blog: http://bradhedlund.comTwitter: http://twitter.com/bradhedlundE-mail: [email protected]
Comments welcome.
BRAD HEDLUND .COM5/26/2010
http://bradhedlund.com/about/
![Page 3: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/3.jpg)
Why Evolve Data Center Switching?
From Connectivity to Virtualization
The Server is a fluid object
The virtual machine is the new Server
The physical machine is the new Network
Miniaturization & Scale
Any server, any VLAN, anywhere, anytime
What’s Next?
The “Data Center” becomes a fluid object
Transformational Paradigm Shifts
New Requirements
Large, Flat, Scalable L2 fabricsSimpler, Smarter NetworksUltra High AvailabilityStorage/Ethernet consolidationMore bandwidth
BRAD HEDLUND .COM5/26/2010
![Page 4: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/4.jpg)
REVISITING CLASSIC ETHERNETWhat works? What needs to be improved?
SW1
SW2 SW3
BRAD HEDLUND .COM5/26/2010
Narrative of this section located at: http://bradhedlund.com/2010/05/07/setting-the-stage-for-trill/
![Page 5: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/5.jpg)
Flooding Behavior
A
B
C
A L L
A
B
C
A L L
A L LSW1
MAC TableA Eth 1/1B Eth 1/2C Eth 1/3
Unknown UNICAST BROADCAST
D
D
D SW1
•Allows Plug & Play•Assists in MAC Learning
•Foundational Behavior•Assists in Discovery•Assists in MAC Learning•Plug & Play
Plug & Play
BRAD HEDLUND .COM5/26/2010
![Page 6: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/6.jpg)
SW3 MAC TableA -- Eth 1B -- Eth 2C -- Eth 3D -- Eth 3
SW1 MAC TableA -- Eth 1B -- Eth 1C -- Eth 2D -- Eth 2
SW4 MAC TableA -- Eth 3B -- Eth 3C -- Eth 1D -- Eth 2
Classical Ethernet MAC Learning
A B C D
SW4SW3
SW1
1 2
3
21
21
3
Useful
Unnecessary
All switches learn all MACs•Plug & Play•Inefficient
Improvement Area (TRILL*)
BRAD HEDLUND .COM5/26/2010
![Page 7: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/7.jpg)
Flooding with Multiple Paths
SW3
SW1 SW2
SW4
A B
No Ethernet TTLInfinite Loop
Improvement Area (TRILL)
BroadcastUnknown UnicastFlooding Required
Plug & Play loop prevention is needed…
BRAD HEDLUND .COM5/26/2010
![Page 8: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/8.jpg)
Classical Ethernet Loop Prevention
SW3
SW2
A B
SW4
SW1
Spanning Tree Protocol (STP)
R
SW3 SW2
SW4
•Plug & Play•Loop Free for flooded traffic
STP Enforced Tree Topology
SW1 R
•Single Topology for all traffic•Single Path for Unicast•Single Path for Multicast •50% Bandwidth Unused•Timer based recovery
Improvement Areas (TRILL*)
BRAD HEDLUND .COM5/26/2010
![Page 9: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/9.jpg)
SCALING THE DATA CENTER NETWORK
Challenges, Approaches,
SW1 SW2
SW3 SW4 SW5 SW6
BRAD HEDLUND .COM5/26/2010
Narrative of this section located at: http://bradhedlund.com/2010/05/07/setting-the-stage-for-trill/
![Page 10: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/10.jpg)
Scaling out Tier 1
Why scale out Tier 1?
•More Bandwidth at Tier 2•Larger Tier 2 Scalability•Smaller Tier 1 switches•Spread Risk (RAID)•Pay as you grow•Lower latency
Tier 2
BRAD HEDLUND .COM5/26/2010
SW3 SW4
SW5 SW1 SW2 SW6
![Page 11: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/11.jpg)
Scaling out Tier 1 with Classic Ethernet
SW1 SW2
A B
SW5 SW6
SW3
R
SW4
Tier 1
Tier 2
Why Scale Tier 1?•More Bandwidth at Tier 2•Larger Tier 2 Scalability•Smaller Tier 1 switches•Pay as you grow•Low latency
•Spanning Tree still required•Blocking on all but one path•Did not gain scalability•Did not gain bandwidth
Improvement Area (TRILL)
with Spanning Tree Protocol (STP)
BRAD HEDLUND .COM5/26/2010
![Page 12: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/12.jpg)
Scaling out Tier 1 with Classic Ethernet
SW1 SW2
A B
SW5 SW6
SW3
R
SW4
Tier 1
Tier 2
Why Scale Tier 1?•More Bandwidth at Tier 2•Larger Tier 2 Scalability•Smaller Tier 1 switches•Pay as you grow•Low latency
•Unbalanced at Tier 2•Only 1 switch with all paths•Did not gain scalability•Did not gain bandwidth
with Spanning Tree Protocol (STP)
Alternate Topology
BRAD HEDLUND .COM5/26/2010
![Page 13: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/13.jpg)
Scaling UP Tier 1 with Classic Ethernet
SW1 SW2
SW3
SW2SW1
SW4
Rigid (2) Switch Tier 1 design constraint forces Scaling UP – not OUT
Improvement Area (TRILL)
BRAD HEDLUND .COM5/26/2010
![Page 14: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/14.jpg)
Multi Path with Classic Ethernet
SW2SW1
SW4SW3
Multi Chassis Ether Channel (MCEC) aka Virtual Port Channel (vPC)
SW3 SW4
STP finds a Loop Free TopologyNo Blocking Paths, All links active
State Synchronization at Tier 1
Not a trivial accomplishment!Several different states must be synced
STP RolesMAC learningLACPInterface states/configSplit Brain detection
R
Improvement area (TRILL)Multi path with minimal state/sync complexity
BRAD HEDLUND .COM
SW1R
5/26/2010
![Page 15: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/15.jpg)
Scaling Tier 2 with Classical EthernetScale Bandwidth or Size? – Pick one, you can’t have both!
SCALING BANDWITH SCALING SIZE
SW5 SW6SW4SW3
SW1 SW2SW2SW1
SW4SW3
OR
Trade-off between Bandwidth or SizeTier 1 switch density key scaling factor
Improvement Area (TRILL)
Tier 1
Tier 2
BRAD HEDLUND .COM5/26/2010
![Page 16: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/16.jpg)
Scaling out Tier 1 with MCEC
SW1 SW2
SW3 SW4
R
Increasingly Complex State Synchronization across (4) or more Tier 1 switches
SW1
SW3 SW4
R
States & Role Sync X 4STP RolesMAC learningLACPInterface states/configSplit Brain detection
Increased Complexity = Difficult & Fragile scaling
STP still present
SW6SW5
BRAD HEDLUND .COM5/26/2010
![Page 17: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/17.jpg)
Scaling out Tier 1 with MPLS
SW1 SW2
Replacing L2 switching with L3 + MPLS and L2 pseudo wire full mesh Overlay via VPLS
N * (N-1) / 2
Increased Complexity = Difficult & Fragile scaling
SW6SW5
IP + MPLS +VPLS
L2 pseudo wires(full mesh required)
SW3 SW4 SW7 SW8 SW9 SW10
Complex overlay of L2 services over L3
MPLS skill sets?Configuration Intense NOT Plug & Play!
Sound complicated? That’s because it is.
BRAD HEDLUND .COM5/26/2010
![Page 18: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/18.jpg)
TRILL - LAYER 2 MULTI PATHINGAn Introduction
SW1
SW2 SW3
BRAD HEDLUND .COM5/26/2010
Narrative of this section to be posted at: http://www.internetworkexpert.org/topic/trill/
![Page 19: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/19.jpg)
Design Goals for TRILL
Switching• Minimal Configuration• Plug & Play• Auto Discovery• Auto Learning• Flat Addressing• Spanning Tree Protocol
(STP)• Slow Convergence• Single Path• Edge-to-Root Rigid
Design• Single Multicast Tree• Constrained Scaleability
TRILL Routing• Configuration Intense• Configured Learning• Configured Discovery• Plan & Play• Fast Convergence• Multiple Paths• Load Balancing• Multiple Multicast Trees• Hierarchical Forwarding• Any-to-any Flexible
Design• Highly Scalable
BRAD HEDLUND .COM5/26/2010
•Minimal Configuration•Plug & Play•Auto Discovery•Efficient MAC Learning•Multiple Paths•Load Balancing•Any-to-any Flexible Design•Highly Scalable•Fast Convergence
The best of Switching and Routing
![Page 20: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/20.jpg)
SW3 MAC TableA -- Eth 1B -- Eth 2C -- Eth 3D -- Eth 3
SW1 MAC TableA -- Eth 1B -- Eth 1C -- Eth 2D -- Eth 2
SW4 MAC TableA -- Eth 3B -- Eth 3C -- Eth 1D -- Eth 2
MAC Learning -- Evolved
A B C D
SW4SW3
SW1
1 2
3
21
21
3
Useful
Unnecessary
All switches learn all MACs•Plug & Play•Flat & Inefficient
BEFORE
Improvement Area (TRILL*)
BRAD HEDLUND .COM5/26/2010
![Page 21: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/21.jpg)
SW3 MAC TableA -- Eth 1B -- Eth 2
SW1 MAC Table
SW4 MAC Table
C -- Eth 1D -- Eth 2
MAC Learning -- Evolved
A B C D
SW4SW31 2
3
21
3
Useful
Hierarchical
SW3 -- Eth 1SW4 -- Eth 2
AFTERC -- SW4
B -- SW3
ConversationLearning
Converstation Based Learning•Plug & Play•Hierarchical, Efficient•Scalable
BRAD HEDLUND .COM5/26/2010
SW121
![Page 22: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/22.jpg)
Multi Pathing -- Evolved
A B
SW5 SW1 SW2 16
MAC TableB -- SW4A -- Eth5
SW3
SW6
SW TableSW4 -- Eth 1,2,3,4
MAC TableA -- SW3B -- Eth5
SW TableSW3 -- Eth 1,2,3,4
SW421 3 4 21 3 4
Per Flow L2-L4 hashing
55
16-way Equal Cost Multi Path (ECMP) Layer 2 Forwarding
BRAD HEDLUND .COM5/26/2010
![Page 23: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/23.jpg)
Loop Free Flooding -- Evolved
A B
SW5 SW1 SW2 16
SW3
SW6
21 3 4
5
A unique loop free forwarding topology for Broadcast & Unknown Unicast ONLY
BRAD HEDLUND .COM5/26/2010
SW421 3 4
5
Does not punish path availability for known unicast & multicast conversations
![Page 24: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/24.jpg)
Multicast -- Evolved
S1
SW5 SW1 SW2 16
SW3
SW6
SW4
Per (S,G) Topologies
More Bandwidth for Multicast – All possible Topologies Used
S2R1 R2
BRAD HEDLUND .COM5/26/2010
![Page 25: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/25.jpg)
Scaling out Tier 1 with TRILL
SW5 SW1 SW2
SW3
SW6
SW4
16
No complicated MCEC state sync No Spanning Tree No complicated MPLS overlay Simple Configuration Flexible any-to-any design
Why scale out Tier 1?•More Bandwidth at Tier 2•Larger Tier 2 Scalability•Smaller Tier 1 switches•Spread Risk (RAID)•Pay as you grow•Low latency
Tier 1
Tier 2
Plug & Play L2 fabric with L3 Scalability and Robustness
BRAD HEDLUND .COM5/26/2010
![Page 26: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/26.jpg)
Scaling Tier 2 with TRILLFlexible design can scale both Bandwidth and Size
SCALING BANDWITH SCALING SIZE
SW2SW1
SW4SW3
AND
Tier 1
Tier 2
SW2 SW3
SW5 SW8SW7
SW4SW1
SW6
Minimal Trade-off between Bandwidth or SizeTier 1 switch density less of a scaling factor
BRAD HEDLUND .COM5/26/2010
![Page 27: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/27.jpg)
Part 1. Summary
BRAD HEDLUND .COM
Addressing scalabilityConversational MAC Learning*Hierarchical MAC Forwarding*
Bandwidth scalability16 active pathsMultiple Multicast Topologies*
Domain Size scalabilityFlexible 16-switch Tier 1 design options
Plug and Play Layer 2 with Layer 3 Scalability and Robustness
Robust L3 characteristicsLink State Topology AwarenessFast Convergence
Plug & PlayAuto LearningSimple Configuration**
* Not present in current RFC 5556 (TRILL)* Present in Cisco specific enhancements, Cisco driving PARs to improve current RFC** (3) NX-OS commands per Nexus switch, subject to change
5/26/2010
![Page 28: Evolving Data Center Switching - Amazon Web Servicesinternetworkexpert.s3.amazonaws.com/2010/trill1/... · Evolving Data Center Switching Part 1. Setting the Stage for Layer 2 Multi-pathing](https://reader034.fdocuments.in/reader034/viewer/2022042305/5ed07339403b1d777903c41d/html5/thumbnails/28.jpg)
STAY TUNED FOR MORE DETAILS…http://bradhedlund.com/feed/
SW1
SW2 SW3
BRAD HEDLUND .COM5/26/2010