Lecture 14: Inter-domain Routing Stability
CS 268 class
March 8th, 2004
(slides from Timothy Griffin’s tutorial and Craig Labovitz’s NANOG talk)
Outline of the Today’s class
• An Introduction to BGP
• BGP and the Stable Paths problem
• Convergence of BGP in the real world
• Conclusions and Open Issues
Inter-domain Routing basics
• Internet is composed of over 16000 autonomous
systems
• BGP = Border Gateway Protocol
– Is a Policy-Based routing protocol
– Is the de facto inter-domain routing protocol of
today’s global Internet
• Relatively simple protocol, but configuration is
complex and the entire world can see, and be
impacted by, your mistakes.
BGP Operations (Simplified)
Establish session on TCP port 179
Exchange all active routes
Exchange incremental updates
AS1
AS2
While connection is ALIVE exchangeroute UPDATE messages
BGP session
Four Types of BGP Messages
• Open : Establish a peering session.
• Keep Alive : Handshake at regular intervals.
• Notification : Shuts down a peering session.
• Update : Announcing new routes or withdrawing
previously announced routes.
announcement = prefix + attributes values
Two Types of BGP Neighbor Relationships
• External Neighbor (eBGP) in a different Autonomous Systems
• Internal Neighbor (iBGP) in the same Autonomous System AS1
AS2
eBGP
iBGP
iBGP is routed (using IGP!)
iBGP Peers Must be Fully Meshed
iBGP neighbors do not announce routes received via iBGP to other iBGPneighbors.
eBGP update
iBGP updates
• iBGP is needed to avoid routing loops within an AS
• Injecting external routes into IGP does not scale and causes BGP policy information to be lost
• BGP does not provide “shortest path” routing
• Is iBGP an IGP? NO!
Important BGP attributes
• LocalPREF– Local preference policy to choose “most”
preferred route
• Multi-exit Discriminator– Which peering point to choose?
• Import Rules– What route advertisements do I accept?
• Export Rules– Which routes do I forward to whom?
Route Selection Summary
Highest Local Preference
Shortest ASPATH
Lowest MED
i-BGP < e-BGP
Lowest IGP cost to BGP egress
Lowest router ID
traffic engineering
Enforce relationships
Throw up hands andbreak ties
Implementing Customer/Provider and Peer/Peer relationships
• Enforce transit relationships – Outbound route filtering
• Enforce order of route preference– provider < peer < customer
Two parts:
Import Routes
Frompeer
Frompeer
Fromprovider
Fromprovider
From customer
From customer
provider route customer routepeer route ISP route
Export Routes
Topeer
Topeer
Tocustomer
Tocustomer
Toprovider
From provider
provider route customer routepeer route ISP route
filtersblock
Outline of the Today’s class
• An Introduction to BGP
• BGP and the Stable Paths problem
• Convergence of BGP in the real world
• Conclusions and Open Issues
This talk
What Problem is BGP solving?
Underlying problem
Shortest Paths
Distributed means of computing a solution.
X?
RIP, OSPF, IS-IS
BGP
• aid in the design of policy analysis algorithms and heuristics, • aid in the analysis and design of BGP and extensions,• help explain some BGP routing anomalies, • provide a fun way of thinking about the protocol
Having an X can
1
Q : How simple can X get? A: The Stable Paths Problem (SPP)
2 5 5 2 1 0
0
2 1 02 0
1 3 01 0
3 0
4 2 04 3 0
3
4
2
1
• A graph of nodes and edges, • Node 0, called the origin, • For each non-zero node, a
set or permitted paths to the origin. This set always contains the “null path”.
• A ranking of permitted paths at each node. Null path is always least preferred. (Not shown in diagram)
An instance of the SPP :
When modeling BGP : nodes represent BGP speaking border routers, and 0 represents a node originating some address block
most preferred…least preferred (not null)
Yes, the translation gets messy!
5 5 2 1 0
1
A Solution to a Stable Paths Problem
2
0
2 1 02 0
1 3 01 0
3 0
4 2 04 3 0
3
4
2
1
• node u’s assigned path is either the null path
or is a path uwP, where wP is assigned to
node w and {u,w} is an edge in the graph,
• each node is assigned the highest ranked
path among those consistent with the paths
assigned to its neighbors.
A Solution need not represent a shortest path tree, or a spanning tree.
A solution is an assignment of permitted paths to each node such that
A Stable Paths Problem may have multiple solutions
First solution
1
0
2
1 2 01 0
1
0
2
1
0
2
2 1 02 0
1 2 01 0
2 1 02 0
1 2 01 0
2 1 02 0
Second solutionDISAGREE
Multiple sets of BGP routing policies can map down to the same Stable Paths Problem : DISAGREE in RPSL (Version I)
1
0
2
1 2 01 0
2 1 02 0
import : from AS1 action pref = 0; accept ANY; from AS0 action pref = 10; accept ANY; export : to AS2 announce ANY;
import : from AS2 action pref = 0; accept ANY; from AS0 action pref = 10; accept ANY; export : to AS1 announce ANY;
export : to AS1, AS2 announce AS0;
DISAGREE in RPSL (Version II)
1
0
2
1 2 01 0
2 1 02 0
import : from AS-ANY action pref = 0; accept community.contains(1:1); from AS-ANY action pref = 10; accept ANY; export : to AS2 announce ANY;
export : to AS1 set community.append(2:1); announce AS0; to AS2 set community.append(1:1); announce AS0
import : from AS-ANY action pref = 0; accept community.contains(2:1); from AS-ANY action pref = 10; accept ANY; export : to AS1 announce ANY;
Assume AS1 and AS2 use “neighbor send-community” command ….
DISAGREE in RPSL (Version III)
1
0
2
1 2 01 0
2 1 02 0
import : from AS-ANY accept ANY; export : to AS2 announce ANY;
export : to AS1 action aspath.prepend(AS0, AS0, AS0); announce AS0; to AS2 announce AS0
import : from AS1 action pref = 0; accept ANY; from AS0 action pref = 10; accept ANY; export : to AS1 announce ANY;
The interaction of all BGP policies is directly represented in SPP
Multiple solutions can result in “Route Triggering”
1
02
3
1 01 2 3 0
2 3 02 1 0
3 2 1 03 0
1
02
3
1
02
3
Remove primary link Restore primary link
1 01 2 3 0
2 3 03 1 0
3 2 1 03 0
primary link
backup link
SPP helps explain possibility of BGP divergence
• BGP is not guaranteed to converge to a stable routing. Policy inconsistencies can lead to “livelock” protocol oscillations.
• See “Persistent Route Oscillations in Inter-domain Routing” by K. Varadhan, R. Govindan, and D. Estrin. ISI report, 1996
Solvable Can Diverge
must convergemust diverge
The SPP view :
2
0
31
2 1 02 0
1 3 01 0
3 4 2 03 0
4
4 2 04 3 0
BAD GADGET : No SolutionWith a BGP-likeprotocol, each nodewill do the best it can, so at least onenode will always have the opportunity toimprove its path. Result : persistent oscillation.
SURPRISE : Beware of Backup Policies
2
0
31
2 1 02 0
1 3 01 0
3 4 2 03 0
4
4 04 2 04 3 0
Becomes BAD GADGET if link (4, 0) goes down.
BGP is not robust : it is not guaranteed to recover from network failures.
PRECARIOUS
1
0
2
1 2 01 0
2 1 02 0
3
4
5 6
5 3 1 05 6 3 1 2 05 3 1 2 0
6 3 1 06 4 3 1 2 06 3 1 2 0
4 3 1 04 5 3 1 2 04 3 1 2 0
3 1 03 1 2 0
As with DISAGREE, this part has two distinct solutions
This part has a solution only when node 1 is assigned the direct path (1 0).
Has a solution, but can get “trapped”
What is to be done?
StaticApproach
Inter-AScoordination
Automated Analysis of Routing Policies(This is very hard).
Dynamic Approach
Extend BGP with a dynamic means of detecting and suppressingpolicy-based oscillations?
These approaches are complementary
Research papers on SPP
“An Analysis of BGP Convergence Properties”Timothy G. Griffin, Gordon Wilfong
“Policy Disputes in Path Vector Protocols”Timothy G. Griffin, F. Bruce Shepherd, Gordon Wilfong
SIGCOMM’99
INFOCOM’00
ICNP ‘99
Model BGP, show static analysis is hard
Define Stable Paths Problem and develop sufficient condition for “sanity”
Dynamic solution based on histories
“A Safe Path Vector Protocol”Timothy G. Griffin, Gordon Wilfong
“Stable Internet Routing without Global Coordination”Lixin Gao, Jennifer Rexford
SIGMETRICS’00
Show that if certain guidelines are followed, then all is well.Rule: Do not forward route advertisements from peers or Providers to other peers or providers.
Outline of the Today’s class
• An Introduction to BGP
• BGP and the Stable Paths problem
• Convergence of BGP in the real world
• Conclusions and Open Issues
Convergence in the real-world?
• [Labovitz99] Experimental results from two year study which measured 150,000 BGP faults injected into peering sessions at several IXPs
• Found– Internet averages 3 minutes to converge after
failover– Some multihomed failovers (short to long
ASPath) require 15 minutes
Problems with Distance Vector
• Distance vector protocols (e.g. RIP) suffer routing table loops – Counting-to-infinity– Routing table loops– Bouncing problem
• BGP uses path vector to “solve” problems seen with RIP and other Bellman-Ford derived protocols
Taming Infinity• Routing Information Protocol (RIP) solved
counting to infinity problem by re-defining infinity.– Added speedups: poison reverse, split horizon,
triggered updates.– Strictly increasing O(N)
• ASPath limits “infinity” to the width of the Internet (an ASPath through all your neighbors)– Monotonically increasing– Upper bound?
BGP Convergence ExampleR
AS0 AS1
AS2AS3
*B R via AS3 B R via AS1,AS3 B R via AS2,AS3
*B R via AS3 B R via AS0,AS3 B R via AS2,AS3
*B R via AS3 B R via AS1,AS3 B R via AS2,AS3
AS0 AS1 AS2
** **B R via 203
*B R via 031 B R via 103
N > 4?
AS1673
AS237
AS5696
AS2497
AS1239
AS6453
AS701
AS2914
AS6461
AS5000
AS6113
AS1
2914 237
1 5696 237
1239 5696 237
2497 5696 237
701 6461 5696 237
6461 5696 237
237
5696 237
5000 237
6113 2914 237
1673 5696 237
6453 1239 5696 237
The Problem with BGP
• If we assume 1. unbounded delay on BGP processing and propagation
2. Full BGP mesh BGP peers
3. Constrained shortest path first selection algorithm
• BGP is O(N!), where N number of default-free BGP speakers
There exists possible ordering of messages There exists possible ordering of messages such that BGP will explore all possible such that BGP will explore all possible ASPaths of all possible lengthsASPaths of all possible lengths
BGP and RIP• RIP precisely monotonically increasing. Can explore
metrics (1…N)• BGP monotonically increasing. Multiple (N!) ways to
represent a path metric of N.
• BGP “solved” RIP routing table loop problem by making it exponentially worse…
2117 5696 2129
2117 1 5696 2129
2117 2041 3508 3508 4540 7037 1239 5696 2129
2117 1 2041 3508 3508 4540 7037 1239 5696 2129
2117 2041 3508 3508 4540 7037 1239 6113 5696 2129
2117 1 2041 3508 3508 4540 7037 1239 6113 5696 2129
BGP Best Case
What is the best we can expect from BGP?What is the best we can expect from BGP? Implementation of MinRouteAdver timer Implementation of MinRouteAdver timer
leads to 30 second roundsleads to 30 second rounds• Time complexity is O(n-3)*30 secondsTime complexity is O(n-3)*30 seconds• State/Computational complexity O(n)State/Computational complexity O(n)• At its best, BGP performs as well as RIP2 At its best, BGP performs as well as RIP2
(but uses exponentially more memory in (but uses exponentially more memory in the process)the process)
MinRouteAdver
• Minimum interval between successive updates sent to a peer for a given prefix– Allow for greater efficiency/packing of updates– Rate throttle
• Applied only to announcements (at least according to BGP RFC)
• Applied on (prefix destination, peer) basis, but implemented on (peer) basis
MinRouteAdver
• 30*(N-3) delay due to creation mutual dependencies. Provide proof that N-3 rounds necessarily created during bounded BGP MinRouteAdver convergence
• Rounds due to– Ambiguity in the BGP RFC and lack receiver loop
detection– Inclusion of BGP withdrawals with
MinRouteAdver (in violation of RFC)
Conclusions
• Internet routing has serious convergence problems– Result 1 [Griffin et al.]: BGP does not satisfy the
stable paths problem.– Result 2 [Rexford et al.]: If every AS follows a set
of guidelines then Internet routing should not have convergence problems.
– Result 3 [Labovitz et al.]: An extensive measurement study shows that Internet convergence can be in the order of several minutes.
Open issues?
• Convergence analysis (lower,upper) bounds are very weak are “worst-worst” case scenarios.
• Can we design a cleaner protocol that has provably good convergence properties?– What about link-state routing?
• Should we really care about convergence?– Routes to popular prefixes are stable [IMC03]
Top Related