Characterizing and reducing route oscillations in the Internet

Characterizing and reducing route oscillations in the Internet

Vivian Elliott, Kenneth J. Christensen*

Department of Computer Science and Engineering, University of South Florida, 4202 East Fowler Avenue, ENB 118, Tampa, FL 33620, USA

Received 2 November 2001; revised 24 April 2002; accepted 1 May 2002

Abstract

Oscillation of routes in the Internet causes unnecessary overhead. Border Gateway Protocol (BGP) transactions collected from the MAE-

East exchange point for 2000 (January–December) show that approximately 16% of routing overhead traffic exhibits oscillating

Autonomous System paths. About 66% of these paths used extra, unnecessary hops to route data traffic resulting in up to 10% extra-hop

count. Common characteristics are shown to exist in oscillating routes. Our findings demonstrate that long-theorized route oscillations really

do occur in the Internet. Faulty implementations and/or poor policy choices are likely causes, where the currently specified method of BGP

implicit withdrawals causes propagation through the Internet. To reduce oscillations, we propose a new method of forcing explicit

withdrawals in BGP. Simulation experiments with forcing explicit withdrawals show an overall reduction of the transaction traffic, as well as

a reduction in path length.

q 2002 Elsevier Science B.V. All rights reserved.

Keywords: Routing; Border Gateway Protocol; Route oscillation; Route flap damping

1. Introduction

The routing protocol used in the Internet is the Border

Gateway Protocol (BGP) [1–3]. BGP is a policy-based,

path-vector protocol that controls data transmission by

providing a traversal path through Autonomous Systems

(AS) in the Internet. A BGP router receives the AS paths to a

particular destination from one or more peers. The length of

the AS path is one of the attributes, that is, considered in

selecting the best path. Other attributes, such as vendor-

specific weights and local preference, can cause the router to

choose a path that is not the shortest path. Routing protocols

that strictly use the shortest path as the criterion for

preferring a particular path have been shown to converge to

a single route [4]. However, BGP has been theoretically

shown to cause diverging routes [5,6], because BGP routing

decisions consider attributes other than path length.

Routing instability has been informally defined as the

rapid change of network reachability and topology infor-

mation [7]. Network outages are responsible for some of the

instability. However, in some cases, the instability is not

caused by an actual outage. Several routes may be available

to a particular destination and the instability can be caused

by a router selecting different routes to a destination at

different times. Routing policies can change over time

resulting in one of those routes becoming unavailable. In

this paper, the term outage refers to any condition in which a

route becomes unavailable. Our goal is to locate instability

in the working Internet and propose methods for reducing

route oscillation. The main contributions of our work can be

summarized as follows:

1. Common characteristics are shown to exist in oscillating

routes. We identify four different types of route

oscillation.

2. Routing oscillations can be reduced by requiring explicit

withdrawals in the BGP protocol and by executing the

decision process for each received route. Simulation

experiments quantify the impact on both overhead and

data traffic.

The remainder of this paper is organized as follows.

Section 2 describes BGP routing instability and related work.

Section 3 describes the techniques used to study the BGP

update transactions and describes a taxonomy for route

oscillation. Section 4 describes the changes proposed to the

BGP standard and demonstrates the impact of these changes

through experimentation. Section 4 also discusses interoper-

ability between the proposed changes and the existing BGP

specification.Section5summarizesandsuggests futurework.

0140-3664/03/$ - see front matter q 2002 Elsevier Science B.V. All rights reserved.

PII: S0 14 0 -3 66 4 (0 2) 00 1 30 -5

Computer Communications 26 (2003) 143–153

www.elsevier.com/locate/comcom

* Corresponding author. Tel.: þ1-813-974-4761; fax: þ1-813-974-5456.

E-mail addresses: [email protected] (K.J. Christensen),

[email protected] (V. Elliott).

http://www.elsevier.com/locate/comcom

2. The BGP protocol and routing instability

BGP peers in separate ASs communicate their respective

routes to each other implementing a path-vector protocol.

Fig. 1 shows ASs interconnected by BGP routers. A multi-

homed AS is one that receives data traffic only for addresses

within the AS. It reaches the Internet via more than one

Internet Services Provider (ISP). In Fig. 1, the multi-homed

AS is connected to ISP 1 using Router 1 and to ISP 2 using

Router 2, so it can send and receive traffic via either ISP.

ISPs connect to the Internet via network exchange points. In

Fig. 1, the network exchange point is accepting traffic from

both ISPs via Router 5 and communicates to the rest of the

Internet via Router 6 (thus, there are two possible paths to

and from the Internet and the potential for oscillating

between the two paths). BGP peers exchange four types of

messages. The open message is the first message sent after a

connection is established. Its purpose is to identify the peers

and agree on protocol parameters, such as timer values. The

update message is the method by which two BGP peers

exchange routing information. A notification message is

used to inform a BGP peer of an error condition. This

usually immediately precedes the termination of the

connection and allows the receiving peer to determine the

cause of the termination. The keepalive message is sent

between BGP peers to confirm that the connection is still

valid. The frequency of keepalive messages is based on

timers that were established by the open message.

Update messages, or routing advertisements, contain two

types of information, announcements and withdrawals of

address prefixes. An address prefix is concise way of

defining a set of network addresses. An announcement

indicates that a particular prefix can be reached via the

advertising peer and a withdrawal is an indication that a

particular address prefix can no longer be reached via the

advertising peer. BGP is designed to send routing adver-

tisements only when there is a topology change and not at

predetermined time intervals as with some other protocols.

Announcements consist of several attributes that can

influence the routing decision. Attributes are weighed at

each step in the decision-making process, hence a tie can

result. This attribute evaluation results in a ‘degree of

preference’ for the route that will be used to compare this

route to other routes for the same destination. While the

evaluation of attributes may vary by vendor, attributes are

usually considered in the following order: next hop validity,

local preference, local route availability, AS path length,

origin, Multi-Exit Discriminator (MED), confederation,

inter-gateway protocol reachability and router identifier. If

the evaluation of the first three attributes results in a tie

between two paths, the AS path length is considered.

Therefore, in those cases, where the first three attributes are

the same, BGP should return the same decision as a shortest-

path protocol [8].

A BGP router maintains three levels of routing tables

called the Routing Information Base (RIB) as shown in Fig.

2. The announcements received from the router’s peers are

stored in a separate inbound RIB for each peer. The AS

policies defined in the router configuration are considered

when determining whether to subject a peer’s announce-

ment that was stored in the inbound RIB to the decision

process. BGP does not go through the decision process to

insert information into this RIB. Instead, when a new route

for an existing prefix is received from a peer, the existing

route for that prefix is considered implicitly withdrawn and

replaced by the new route. The local RIB contains the best

routes that have been chosen from the inbound RIBs by

going through the BGP decision process. This is the

information used to forward data, and there is only one

local RIB per router and only one path to a destination in

this RIB. If one of these paths becomes unavailable by a

withdrawal advertisement, the router returns to the inbound

RIBs to replace that route. If no other route to that

destination can be found, the destination is considered

Fig. 1. Inter-AS communication.

Fig. 2. RIBs and BGP policy, decision, and advertisement.

V. Elliott, K.J. Christensen / Computer Communications 26 (2003) 143–153144

inaccessible. The outbound RIBs contain the announce-

ments made to each peer and, like the inbound RIBs, are

maintained separately for each peer. This allows the router

to announce different paths to different peers depending on

the policy it has with that particular peer. A router cannot

use a route it has not received, and it will not announce a

route that it is not using.

The relationship between BGP policies and router

instability has been previously studied. Govindan et al. [9]

suggest a routing registry that would permit an AS to

determine whether its policies were in conflict with those of

a peering AS. An example is cited, where three ASs could

independently set up a cycle of dependencies on each other.

Such a conflict would lead to a continuous exchange of

routing information. Currently, there are two such routing

registries available on the Internet, RIPE [10] and APNIC

[11]. RIPE is an organization of European Internet providers

who share their policy information through the Routing

Arbiter Database. APNIC is a similar organization for Asia-

Pacific Internet providers. Fig. 3 shows RIPE entries for the

policies that AS 1755 has with peering ASs. For example,

the first ‘as-in’ line shows that AS 1755 will accept any

announcement from AS 1 assigning it a cost of 200. A lower

cost represents a more desirable route. Problems with

routing registries include the size of the registry and the

staleness of the information contained therein. In addition,

routing registries rely on the cooperation of the ASs, many

of which may not be willing to disclose their policies.

Other researchers have described methods that require

less cooperation among ASs than routing registries do.

Kamizuru and Amagai [12] and Fujinaga et al. [13] suggest

monitoring the BGP update transactions as they arrive at the

AS. In Ref. [12], the routing transitions for a particular

destination are grouped by route and analyzed for

instability. The analysis criteria can be specified by the

system user. For example, a notification may be requested if

a particular route has changed more than a given number of

times in a period. Then, the source and cause of the

instability is determined with the intent of correcting the

problem by changing the implemented policies at the source

AS. In contrast, Ref. [13] implements a message log for later

manual analysis.

Gao and Rexford [14] explore a solution that constrains

the policies that an AS is allowed to implement. They

perceive the Internet as a hierarchy of providers, clients, and

peers. Different policy guidelines are suggested for the

relationships that can exist among these groups of ASs.

Similarly, Griffin et al. [15] suggest that certain BGP

policies and corresponding AS paths exist that are

guaranteed not to cause routing instability. They define

the Simplified Path-Vector Protocol (SPVP) as a framework

for any path-vector protocol, like BGP, to follow. These safe

AS paths are ranked to allow the selection of the best option.

Paths that could cause instability are prohibited.

Route flap damping [16] is widely used in the working

Internet to reduce routing instability. Like routing policies,

route flap damping is another way for an AS to determine

whether to advertise routes received from a peering AS to

other peers. It stores a penalty value with each route, that is,

incremented when the route changes, or flaps, and it

decrements the penalty value over time. When the penalty

value exceeds a threshold defined by the AS, the AS will no

longer advertise the route. This makes the destination

unavailable until its route stabilizes (thus, route flap

damping can cause outages to occur while route advertising

is blocked). Since the penalty value will decrease over time

when the route is stable, eventually, the AS will advertise

the route again.

2.1. Previous work in analysis of Internet routing data

Chinoy [17] collected routing update information logged

by NSFNET routers with the goal of quantifying routing

information changes in the Internet. The traffic was

described in terms of connectivity transitions, unreach-

ability cycles, and clusters. A connectivity transition is a

change in the reachability of a destination; an unreachability

cycle is the period of time that a destination is unavailable;

and a cluster is a group of destinations that undergo

unreachability cycles together. It was shown that less than

3% of clusters consisting of less than 10 networks produced

the most connectivity transitions. Paxson [18] investigated

the stability of routes from the source to the destination

using the traceroute facility to measure the path between

two Internet sites. In this case, stability was defined in terms

of prevalence and persistence. Prevalence is the condition,

where a particular route in existence at a point in time would

be available at a future time. Persistence is the length of time

that it would be available. This investigation showed that

while a particular route would usually be available in the

future, the length of time the routes persisted varied widely

from seconds up to days with two-thirds of the routes

persisting for days or weeks. Govindan and Reddy [19]

developed a snapshot of the Internet topology using routing

update data gathered at a large Internet service provider and

at one of their large exchange point. They found that as

more routes became available, routing instability increased

making it difficult to create a precise topology model of the

Internet.

Labovitz et al. [7] analyzed BGP update data received at

several of the major US network exchange points. In the

study, they found that the same routes were advertised or

withdrawn with no true change in topology. One Internet

service provider announced 259 unique prefixes, but

Fig. 3. Snippet from a RIPE-181 database.

V. Elliott, K.J. Christensen / Computer Communications 26 (2003) 143–153 145

generated over 2.4 million withdrawals for 14,112 prefixes.

In a subsequent paper, Labovitz et al. [20] analyzed the

sources of the redundant information. Much of the

instability was caused by problems in router vendor

software or in router configuration. Some vendors

implemented a stateless form of BGP in which the router

did not recall which prefixes were sent to which peers. Upon

receipt of any topology change, the routers would send

withdrawals to all peers regardless of whether they had

previously received an announcement. In addition, some

routers were configured to advertise changes in routing

characteristics that only affected routing within the AS. This

information should not have been contributing to the inter-

AS BGP traffic.

In this paper, BGP update transactions are examined to

locate routing oscillations in the working Internet as

described in Ref. [5]. In Ref. [5], Varadhan et al. describe

a set of theoretical conditions that would produce a

persistent oscillation of routing path data. The conditions

are such that a router would first prefer one route, then

another, then prefer the original route again. We describe

the different types of oscillation found in the Internet and

explore the performance impact of the additional announce-

ments and the routing transitions. Our work differs from

previous papers in that the oscillating routes are isolated for

study and a taxonomy is proposed.

3. Characterization of BGP transactions

Routing advertisements were chosen for this study,

because they describe all routing changes including the

short-lived changes. In addition, a router will never

advertise a route that was not inserted into its forwarding

table, so any advertised routes existed on the announcing

ASs forwarding table for some period. BGP routing

advertisements are collected by the Internet Performance

Measurement and Analysis (IPMA) project [21], a joint

effort between the University of Michigan and Merit

Network, Inc. The advertisements are collected at the

major US network exchange points. For this study, data

from MAE-East, the largest exchange, was used. MAE-East

is the MCI WorldCom facility where ISPs connect to each

other to exchange Internet traffic [22]. The collected data

was translated to a text format using software from the

Multi-threaded Routing Toolkit (MRT) [23]. A sample of

the BGP transaction data after this translation is shown in

Fig. 4. From left to right, the fields are type of data,

timestamp in seconds since January 1, 1970, update type

(announcement or withdrawal), IP address of the announ-

cing router, AS number of the announcing AS, announced

prefix, AS path, source of the update (external or internal),

next hop, local preference, MED, community, aggregator,

and atomic aggregator. The MED indicates the preferred

path, when two ASs are linked by more than one BGP

peering router. The Community attribute provides a way for

the AS to group routes that have similar policies. The

Aggregator attribute identifies the AS and router that

consolidated prefixes received from different peers into a

single aggregated prefix. This attribute will appear on

announcements for the aggregated prefix. The Atomic

Aggregator is a flag that indicates that some information

was lost when an AS chose to aggregate prefixes. For

example, an AS may choose to aggregate, or consolidate,

prefixes that contain differences in other attributes.

For this study, 24 h of data were extracted at 1-month

intervals from January 2000 to December 2000 resulting in

over 6.7 million advertisements. Weekend days were not

included to prevent weekday/weekend variability from

influencing the study. Oscillating routes were defined as

prefixes received from a particular AS with more than one

unique AS path attribute with the same value in the local

preference attribute. The advertisements for the oscillating

routes were extracted into individual files and sorted by their

timestamps to provide a chronological log of the activity on

that route for the day. The summary of this characterization

is shown in Table 1. Total advertisements show the number

of transactions passed from the peering ASs to MAE-East

for the day. The unique AS/prefix statistic is the number of

AS/prefix combinations announced to MAE-East. For

Fig. 4. BGP transaction data.


example, if AS 1 announced prefix 12.10.175.0/24 and later

AS 1239 announced the same prefix, this would constitute

two occurrences. However, if AS 1 announced the prefix at

10:00 PM and again at 10:15 PM, this would be a single

occurrence of this statistic. The total advertisements

exhibiting oscillations are the number of advertisements

generated by prefixes with oscillating routes. The unique,

AS/prefixes-exhibiting-oscillations statistic shows the num-

ber of AS/prefix combinations involved in the oscillating

advertisements. There does not seem to be a pattern to the

data. Increased total announcements do not always result in

increased values of the other statistics. By month, the

oscillating announcements account for between 8 and 29%

of the overall traffic. However, the number of unique

oscillating prefixes accounts for between 5 and 8% of the

unique announcements. Since BGP should only be sending

transactions for changing topology, this represents an

opportunity to reduce BGP traffic by up to 21%. This is a

significant reduction in control, or overhead, traffic in the

Internet.

3.1. Routing oscillation taxonomy

Oscillations can be categorized into one of four general

types and are shown in Fig. 5. The four types of oscillation

are (1) basic oscillation, (2) repeater oscillation, (3) extra-

hop oscillation, and (4) two (or more) originating ASs. The

basic oscillation involves a variation of a single AS number

in the path between two or more different ASs. In this type

of oscillation D, the destination AS, may announce its routes

to two different ASs shown as A and B in the diagram. X

receives announcements from both A and B. In this case, the

AS where a faulty routing decision is made is X. It has the

path A–D in its routing table when it receives the B–D

announcement. Since the announcement is coming from two

different ASs, the BGP decision process is invoked, and it

selects route B–D. Then, when X receives the announce-

ment A–D again, the selection process chooses A–D. This

condition was identified in the data as those paths containing

exactly three ASs with the second AS varying between two

values, and it occurred in 8% of the prefixes in our data.

In repeater oscillation, one or more of the AS numbers in

the AS path is repeated. Repeater oscillation does not reflect

any change in topology. BGP allows an AS to insert its

number into the AS path attribute multiple times. Since one

of the criteria of the BGP decision-making process is the

length of the AS path, this flexibility allows the AS to

influence that decision. In this case, the AS, that is,

generating the oscillation is D. D sends the route to A,

where the route is installed on A’s inbound RIB for D. Then,

D will send the path D–D to A. AS A recognizes this as an

implicit withdrawal of the original path and installs the new

path. Then, D sends the original single D path to A again.

This type of oscillation was identified in the data when the

Table 1

Summary of BGP transaction characterization

Statistic January February March April May June

Total advertisements 982,239 705,081 525,876 1,317,924 404,366 846,619

Unique AS/prefixes 93,338 96,931 104,635 108,376 80,588 101,109

Total exhibiting oscillations 180,066 203,874 114,326 154,326 44,584 84,167

Unique exhibiting oscillations 4700 6729 8508 6625 4346 6267

Statistic July August September October November December

Total advertisements 762,640 620,056 80,103 73,171 276,838 118,866

Unique AS/prefixes 96,371 83,536 10,145 17,936 105,110 33,207

Total exhibiting oscillations 170,501 186,274 11,060 9056 18,612 9408

Unique exhibiting oscillations 5623 4513 659 556 831 594

Fig. 5. Types of oscillation.


varying AS paths had different numbers of ASs in the path

but had the same number of unique AS values in the path.

Repeater oscillation was present in 66% of the prefixes.

Extra-hop oscillation occurs when an announcement is

made by an AS, A, to two different peers, B and X. B also

announces the path to X. Assuming all other BGP attributes

are equal, X should always select the route from A, because

the AS path is shorter. This type of oscillation has the

greatest impact on flow of data traffic, because it will be

forced to pass through one or more additional ASs while the

longer path is in effect. Extra-hop oscillation existed when

the changing AS paths had different numbers of unique AS

values. It occurred in 60% of the oscillating paths.

The final type of oscillation, two originating ASs, is seen

when a particular prefix, called a multi-homed prefix, is

announced by two or more different ASs. The receiving AS

should choose one of the originating ASs consistently. This

type of oscillation was identified in the data when different

AS paths for a prefix had different originating AS numbers,

and it occurred in 3% of the prefixes. The percentages of

occurrence do not add up to 100%. Some prefixes exhibited,

for example, both extra-hop and repeater oscillation within a

24-h period.

4. The significance and reduction of routing oscillations

While a change in the AS path of a prefix may indicate an

actual network outage, the routing problems archive [24]

did not indicate problems with most of the prefixes in the

study data. For example, for the May data no routing

problems (such as known routing failures) were reported in

the archive, but there were over 4000 oscillating routes. In

addition, repeater oscillation paths have the same AS

numbers in all oscillating paths so there can be no outages at

the ASs contained in the path. Another indication that the

oscillation may not be the result of an outage is the

persistence of oscillation for particular prefixes. Table 2

shows that at least 40% of the oscillating prefixes for a

particular day continued into the next month for six of the

months studied and that 100 prefixes exhibited persistent

oscillations for half the duration of the study (6 months).

When there is no outage, extra-hop oscillation causes

data traffic to be routed through additional unnecessary ASs.

To quantify the effect of the additional hops, a metric was

developed that compares the ideal routing condition to the

actual situation for a particular day. This pathtime ratio, P,

is calculated,

P ¼1

NMTM

� � Xall paths

NATA ð1Þ

where NA is the actual number of hops in a path for the

prefix for a time TA and NM is the minimum number of hops

for the prefix for a time TM: The sum of TA values equals

TM: The numerator reflects the actual data transmission for a

prefix when it is taking both the minimum number of hops

and more than the minimum number of hops. The

denominator reflects the ideal minimum path length case

for the prefix under consideration. For example, if the

minimum AS path length for a prefix is 3 hops and the study

time is one day, then the denominator for that prefix is

3 £ 86,400 s or 259,200. Suppose that for 2 h during the

course of the 24-h period the prefix used an AS path that was

5 hops long, for 1 h it used a path that was 4 hops long, and it

used the minimal path for the other 21 h. Then, the value of

the numerator would be calculated as 5 hops £ 7200 s for

the 5-hop period added to 4 hops £ 3600 s for the 4-hop

period added to 3 hops £ 75,600 s giving 277,200. The

pathtime ratio for this prefix would be 1.070 indicating that

approximately 7% additional bandwidth was used for data

transmission to that prefix for the day. To determine the

overall pathtime ratio for a period, we sum for all prefixes

for the period. Table 3 shows the overall pathtime ratios for

our collected data. The best performance during the period

occurred when actual exceeded ideal by only 0.5% (this

occurred in October). During the worst month (June), actual

exceeded ideal by 10.7%. The shortcoming of the pathtime

Table 2

Oscillating prefixes across month

Date Total prefix Prior month First month

February 4700 1477 1477

March 6729 1633 603

April 6625 2025 381

May 4346 1312 220

June 6267 1757 100

July 5623 2272 46

August 4513 1153 41

September 659 152 0

October 556 97 0

November 831 83 0

December 594 123 0

Table 3

Pathtime ratios

Date Ideal Actual Ratio

January 1,021,852,800 1,107,792,861 1.0841

February 1,298,419,200 1,403,309,344 1.0808

March 1,960,588,800 2,019,078,110 1.0298

April 1,323,129,600 139,8805,967 1.0572

May 932,083,200 948,290,448 1.0174

June 1,562,889,600 1,730,286,589 1.1071

July 1,033,084,800 1,068,410,748 1.0342

August 880,416,000 939,132,435 1.0667

September 67,132,800 67,639,765 1.0076

October 68,947,200 69,296,798 1.0051

November 65,491,200 66,777,474 1.0196

December 86,227,200 89,545,702 1.0385

Total 10,300,262,400 10,908,355,241 1.0590


metric is that it assumes that all hops in an AS path are

equal.

4.1. Reduction of oscillation

Vendor implementations of BGP often default to a

shortest-path algorithm [8]. In this case, none of the extra-

hop or repeater paths would have been selected by the BGP

decision process. However, BGP recognizes implicit route

withdrawals [2], as well as withdrawal advertisements, or

explicit withdrawals. An AS assumes that a path has become

unavailable when a new path for a destination is received

from a particular peer even when a withdrawal advertise-

ment was not sent. This is an implicit withdrawal. The

implementation of implicit withdrawals reduces the number

of transactions sent between peers, because it eliminates the

need for a withdrawal transaction. Further, a peer will have

only one route stored on its forwarding table at any point in

time so it can be assumed that the peer is no longer using the

original route. Therefore, a faulty routing decision made

within an AS could result in a peering AS assuming a better

route is no longer available. There was evidence of implicit

withdrawals in the data examined. For example, in March,

there were 8508 prefixes with changing AS paths. Of these,

2928 (34%) did not contain any withdrawal advertisements,

and March was the month that exhibited most of these

explicit withdrawals.

Overall, during the entire year 88% of the oscillating

prefixes fluctuated between two unique AS paths. Three

different paths were announced for 9% of the cases and four

paths for 2% of the cases. Infrequently, more paths occurred

with one prefix using 31 different paths during a 24-h

period. A comparison shows that the number of unique AS

paths is much lower than the number of prefixes. For

example, in March there were 8508 oscillating prefixes but

only 4513 unique AS paths. Considering that each one of

those oscillating prefixes used at least two unique AS paths,

this means that an average of four prefixes share the same

AS path. Finally, when these unique paths were studied,

several ASs appeared repeatedly in the paths. In the 4513

unique paths for March, there were only 1810 unique AS

numbers. This implies that a minority of the ASs is

generating the oscillating prefixes. Table 4 summarizes

the results for all data.

4.1.1. Proposed changes to the BGP algorithm

BGPs recognition of implicit withdrawals may have been

designed to reduce the additional traffic that would be

caused by explicit withdrawal transactions. However,

announcements are frequently received for replacement

routes when it is unlikely an actual outage occurred. The

evidence is a subsequent re-announcement of the original

route shortly after the announcement of the replacement

route, sometimes in less than 5 s. The data also show that if

these replacement routes had been subjected to the BGP

decision process, they would not have been chosen. We

propose the following change to BGP operation to reduce

route oscillation:

Withdrawals should only be recognized by an explicit

withdrawal advertisement. When a peer sends a replace-

ment route for an existing route without the intervening

withdrawal, the new route should be subjected to the

BGP decision process before acceptance.

Existing BGP implementations process announcements

received from a peering AS as follows:

1. Receive a route announcement from a peer.

2. If a route exists for that destination on the inbound RIB

for that peer, replace it.

3. Invoke the BGP decision process to determine the best

route for the prefix from among the inbound routing

tables.

4. Install the preferred route in the local RIB.

The proposed new BGP explicit withdrawal method

processes announcements as follows:

1. Receive a route announcement from a peer.

2. If a route exists for that destination on the inbound RIB

for that peer, invoke the BGP decision process to

determine which route is the best route received from

that peer.

3. If the received route is better than the existing route,

replace the route.

4. If the route was replaced, invoke the BGP decision

process to determine the best route for the prefix from

among the inbound RIBs.

5. Install the preferred route in the local RIB.

In the proposed method, the decision process at the

local RIB will only be invoked if the route was replaced

on the inbound RIB. Consequently, a second decision

processing cost will be incurred only when a better

route was received from the peer. The BGP

Table 4

Unique AS paths and AS numbers

Date Total prefix Unique path Unique num

January 4700 2117 1023

February 6729 2847 1243

March 6625 4513 1810

April 6625 3158 1291

May 4346 2378 1173

June 6267 3019 1504

July 5623 2816 1346

August 4513 2560 937

September 659 422 313

October 556 318 246

November 831 601 363

December 594 380 308


specification states that withdrawals can be issued for

the following:

1. Recognize that an outage has occurred for a route.

2. If there is no replacement route, issue a withdrawal

advertisement.

3. If there is a replacement route, issue an announce-

ment advertisement that will implicitly withdraw the

existing route on the AS receiving the announcement.

The proposed BGP change for issuing withdrawals can

be summarized as follows:

1. Recognize that an outage has occurred for a route.

2. Issue withdrawal advertisement for the unavailable route.

3. If there is a replacement route, issue an announcement

advertisement for the replacement route.

Note that the two methods of issuing withdrawals will

have the same result on the receiving peer, because an

outage has occurred. However, with the proposed method an

explicit withdrawal advertisement is required when an

actual outage has occurred.

The current BGP specification does not require a

withdrawal advertisement if a path has to be replaced by

another path. Instead, it allows the advertising peer to send

an announcement for the replacing route and implicitly

withdraws the route. The receiving peer will replace the

existing route on the inbound RIB. Under the proposed

method, a route will only be replaced on the inbound RIB if

an explicit withdrawal transaction is received for the route

or if the decision process determines that a better route has

been received. This additional withdrawal advertisement

will only be sent if the existing route is no longer available.

Fig. 6 demonstrates this situation. Suppose D announces a

prefix to both A and B. Router A announces the prefix to X,

and B announces the prefix to C, which announces the prefix

to A. At this point, an invalid routing decision is made in A

if it determines that the path announced by C is better (for

this example, we assume that least hops is better). Router A

installs the route on its local table and announces the route to

X. Under the current implementation of BGP, X will replace

the route A–D with the route A–C–B–D, because it is

assumes that the A–D route is implicitly withdrawn. Under

the explicit withdrawal method, X will subject the received

route to the BGP decision process, since the A–D route was

not explicitly withdrawn. Note that this method will not

prevent the faulty routing decision by A. Faulty implemen-

tations of BGP and/or poor explicit policy choices are likely

the culprit(s) for the observed oscillations. However, our

method will prevent bad decisions from being propagated

throughout the Internet. The observed route oscillations are

based on observed data, these route oscillations should not

be occurring given correct BGP implementation.

4.1.2. Experiments based on the proposed change to the

BGP algorithm

Experiments were conducted on the BGP update

transactions from March to August to determine the impact

of using explicit withdrawals. The experiments were limited

to those prefixes that had only two different AS paths due to

the complexity of identifying oscillation sequences for an

infinite number of AS paths. This eliminated, at most, 15%

of the prefixes announced in a given 24-h period. Table 6

shows a series of transactions for a prefix in the study data

that had paths resembling the configuration of the network

in Fig. 6. The identifier sequence number provides a way to

track the advertisements through the experimentation. Note

that A–D is announced first followed by an announcement

for A–C–B–D, which is assumed to be an inferior route,

followed by another announcement for A–D. For the

experiments, the elapsed time between the announcement of

the inferior route and the subsequent re-announcement of

the preferred route is used to determine whether an actual

outage existed. If the elapsed time is greater than a minimum

assumed outage time (minimum outage time is a control

variable for our experiments) passed into the experiment, then

the announcement for the inferior route is assumed to reflect an

actual outage. In this case, the explicit withdrawal method can

be simulated by generating an explicit withdrawal transaction

followed by the announcement for the inferior path and the

Fig. 6. Sample network.

Table 5

Sample transactions

Identifier Timestamp Type AS path

1 952,708,683 Announcement A–D

2 952,711,539 Announcement A–C–B–D




6 952,712,806 Withdrawal


Table 6

Sample transactions with minimum outage time 15 s

Identifier Timestamp Type AS Path


New ? Withdrawal



New ? Withdrawal






re-announcement for the preferred path. Conversely, if the

elapsed time is less than the minimum outage time, neither

the inferior route nor the preferred route is announced

because the preferred route is the route that would have

remained in place. When an explicit withdrawal transaction

is encountered in the data, an outage is assumed, and the

next announcement is made. Note that the elapsed time

between transactions 2 and 3 in Table 5 is 26 s. Tables 6 and

7 show the results for explicit withdrawal for the

transactions of Table 5. Table 6 shows the results if the

minimum outage time is set to 15 s. The first advertisement

(sequence number 1) is always made to start the process. To

evaluate whether advertisement 2 for the inferior route

should be made, the elapsed time between advertisements 2

and 3 is compared to the minimum outage time. Since 26 s

exceeds 15 s, an outage is assumed. A new explicit

withdrawal advertisement is issued followed by advertise-

ments 2 and 3. A timestamp cannot be assigned to the

withdrawal transaction, because it cannot be known when

the actual outage occurred. A similar situation occurs

between advertisements 4 and 5 with the same results.

Advertisement 6 is an explicit withdrawal in the study data,

and the experiment assumes that this represents a true

outage. That advertisement is accepted along with the

following announcement. Table 7 shows the resulting

transactions with the minimum outage time set to 30 s.

For this case, since the elapsed time of 26 s is less than the

minimum outage time of 30 s, the data does not indicate a

true outage. Advertisement 2 is not generated because this is

an inferior route that would not have been chosen in step 2

of the proposed algorithm. Advertisement 3 would not have

been generated because it represents an unnecessary

advertisement of the existing route. The same situation

occurs for advertisements 4 and 5. As in Table 6, the explicit

withdrawal in advertisement 6 indicates a true outage to the

experiment. The minimum outage time has no impact on the

handling of explicit withdrawal advertisements in the study

data.

Additional experiments were performed with minimum

outage times of 5, 60, 120 and 300 s. The results are

summarized in Table 8 for March–August. In all cases, the

explicit withdrawal method eliminated more announce-

ments than the number of additional explicit withdrawals it

generated resulting in fewer overall BGP transactions. The

pathtime ratio was calculated for each day for those prefixes

that contained extra hops in the inferior AS path. For

comparative purposes, the pathtime ratios from Table 3 are

repeated in the table. With only two exceptions, the

pathtime ratios are closer to 1.0 than the original pathtime

ratios. The exceptions are in the July data for the minimum

outage times 5 and 60. Since the pathtime ratio did improve

for minimum outage times 120 and 300 s for that month, it

can be assumed that the majority of the extra-hop paths were

in effect for more than 60 s. This apparent decline in

Table 7

Sample transactions with minimum outage time 30 s

Identifier Timestamp Type AS path




Table 8

Results of explicit withdrawal method experiments for March–July data

Date Minimum outage time Removed announcement Additional withdrawal Pathtime ratio Original pathtime ratio

March 5 31,818 9359 1.0251 1.0298

60 41,528 2653 1.0238

120 42,593 1944 1.0235

300 43,468 1295 1.0232

April 5 60,347 10,435 1.0270 1.0572

60 72,623 2104 1.0267

120 73,705 1454 1.0265

300 74,725 813 1.0262

May 5 11,363 5883 1.0140 1.0174

60 17,158 1821 1.0135

120 18,159 1180 1.0135

300 18,714 839 1.0134

June 5 39,423 6621 1.0334 1.1071

60 45,458 2607 1.0329

120 47,296 1564 1.0329

300 47,690 1323 1.0328

July 5 42,870 21,095 1.0480 1.0342

60 53,561 14,887 1.0469

120 64,140 9196 1.0301

300 69,642 6412 1.0299


performance occurred because only those prefixes with two

oscillating routes were evaluated for the experiment. The

lower minimum outage times did not remove sufficient

oscillation to produce a positive result against the smaller

base of prefixes. A problem occurs when the last

announcement is an inferior path. Because there is no

subsequent re-announcement of the preferred path to use to

compare the elapsed time, it cannot be determined whether

this final announcement should reflect an outage. This

means that the removed announcements value may be

overstated by, at most, the number of unique prefixes. Since

the highest number of unique prefixes was 8508 for the

study period, this has little effect on the overall results.

A final experiment was conducted to determine the effect

of route flap damping on the transactions produced by the

current method and those produced by the explicit with-

drawal method. In route flap damping, an AS determines

whether it should advertise an oscillating route by assigning

a penalty value to the route. If it is determined that the route

should not be advertised, the destinations represented by the

oscillating route become unreachable by the peers of the

advertising AS. For this experiment, the transactions created

by the simulation using a minimum outage time of 5 s were

chosen because this represents the worst performance by the

explicit withdrawal method. A route was damped if it

oscillated more than five times in 30 min. The results of this

experiment for March–August are shown in Table 9. The

total prefixes are again just those prefixes that have exactly

two oscillating routes. It is evident that the reduced number

of transactions produced by the explicit withdrawal method

results in fewer routes being damped and, as a result,

reducing the number of unreachability conditions in the

Internet. This improvement in reachability is significant.

4.2. Interoperability issues

The major interoperability issue exists where the router

that implements the New Algorithm (the NA router) is

expecting an explicit withdrawal transaction and a peer that

does not implement the new algorithm (the Original

Algorithm or OA router) will continue sending implicit

withdrawals in the form of a new announcement. This could

cause the NA router to retain a path, that is, no longer valid

for evaluation purposes. However, this will not cause a loss

of data. There is still a path through the OA peer because it

sent a new path. It may just not be the best path, since the

NA router did not withdraw the original path and subject the

new path to the decision-making process. Here is an

example. Suppose the NA router has two peers, A and M

that implement the old algorithm. Both peers send a path for

a destination Z. A sends the path A–B–C–Z. M sends the

path M–N–O–P–Z. Assume that the NA router prefers A’s

path because it is shorter. Now A implicitly withdraws the

path by sending A–D–E–F–G–Z because, for example,

router B went down. The new algorithm router rejects the

path because (1) no explicit withdrawal was sent indicating

that A–B–C–Z was unavailable and (2) A–D–E–F–G–Z

is longer than the original path. Because the path has not

changed, the NA router continues to use A as its next hop

when sending data to Z even though a better (shorter) path

exists through peer M. The cost of this usage of suboptimal

paths needs to be investigated. More work needs to be done

on interoperability. For example, some means of peer

routers identifying what algorithms they use and agreeing

on the lowest common denominator may work well.

5. Summary and future work

We have shown that persistent oscillations exist in the

Internet. The oscillations were categorized into one of four

basic types (basic, repeater, extra-hop, and two originating

ASs) based on characteristics of the varying AS path

attributes. The effect of diverging BGP routes was

quantified in terms of wasted bandwidth for the resulting

routes. Wasted bandwidth occurs when extra and unnecess-

ary hops are taken as of a result of a route oscillation. A

metric of the amount of time that additional hops are taken,

called the pathtime ratio, was developed to quantify the

impact on bandwidth waste. The best monthly performance

in the 1-year study period provided a pathtime ratio of

1.005; the ratio for the worst month was 1.107 (i.e. over

10% of observed traffic was taking extra and unnecessary

hops resulting is wasted bandwidth). We found that the

greatest impact to bandwidth waste was caused by one

particular type of oscillating route (extra-hop oscillation)

characterized by using an AS path longer than the minimal

AS path for some period.

While the initiation of oscillation may be a function of

configuration and transaction attributes, the propagation of

that oscillation is a function of the implicit withdrawals that

the BGP protocol allows. We propose that BGP not

Table 9

Results of route flap damping experiments for March–July data

Date Total prefixes Routes damped under old method Routes damped using explicit withdrawals

March 7230 3088 1706

April 5601 2415 1922

May 3891 1067 759

June 5658 1101 883

July 5009 3500 2757


acknowledge implicit withdrawals, since it allows faulty

routing decisions made within an AS to be sent to peering

ASs without the benefit of the BGP decision process at the

receiving AS. We also propose that BGP should force

explicit withdrawal notification and that only these notices

be recognized for a route withdrawal. Explicit withdrawals

were shown to reduce the number of BGP messages,

improve the pathtime ratio, and reduce network unreach-

ability situations if route flap damping is also used.

Interoperability problems are limited to possible selection

of a suboptimal path due to a new algorithm router retaining

a path, that is, no longer valid for evaluation purposes. To

further validate the proposed changes and to further evaluate

interoperability, the new algorithm should be implemented

and tested in a laboratory setting.

Acknowledgment

The authors acknowledge Jennifer Rexford for her

assistance in helping us to better understand BGP operation.

References

[1] B. Halabi, Internet Routing Architectures, New Riders Publishing,

Indianapolis, 1997.

[2] Y. Rekhter, T. Li, A Border Gateway Protocol 4 (BGP-4), RFC-1771,

1995.

[3] J. Stewart, BGP4: Inter-Domain Routing in the Internet, Addison-

Wesley, Reading, MA, 1999.

[4] D. Bertsekas, R. Gallagher, Data Networks, Prentice-Hall, Upper

Saddle River, NJ, 1992.

[5] K. Varadhan, R. Govindan, D. Estrin, Persistent route oscillations in

inter-domain routing, Technical Report 96-631, Computer Science

Department, University of Southern California, 1996.

[6] T. Griffin, G. Wilfong, An analysis of BGP convergence properties,

Proceedings of ACM SIGCOMM (1999) 277–288.

[7] C. Labovitz, G. Malan, F. Jahanian, Internet routing instability, IEEE/

ACM Transactions on Networking 6 (5) (1998) 515–528.

[8] C. Labovitz, R. Wattenhofer, S. Venkatachary, The impact of internet

policy and topology on delayed routing convergence, Technical

Report MSR-TR-2000-74, Microsoft Research, 2000.

[9] R. Govindan, C. Alaettinoglu, G. Eddy, D. Kessens, S. Kumar, W.

Lee, An architecture for stable, analyzable internet routing, IEEE

Network 13 (1) (1999) 29–35.

[10] Reseaux IP Europeens (RIPE) Network Coordination Centre (2000),

URL: http://www.ripe.net/.

[11] Asia Pacific Network Coordination Center (APNIC), URL: http://

www.apnic.net/.

[12] Y. Kamizuru, Y. Amagai, Distributed inter-AS route monitor—

Distributed internet route eye (DIRE), Proceedings of International

Conference on Parallel and Distributed Systems (1998) 533–540.

[13] M. Fujinaga, R. Hotta, T. Asami, A tool for monitoring routing

information in the internet, Proceedings of the IEEE International

Workshops on Parallel Processing (1999) 274–279.

[14] L. Gao, J. Rexford, Stable internet routing without global coordi-

nation, Proceedings of ACM SIGMETRICS (2000) 301–317.

[15] T. Griffin, F. Shepherd, G. Wilfong, Policy disputes in path-vector

protocols, Proceedings of International Conference on Network

Protocols (1999) 21–30.

[16] C. Villamizar, R. Chandra, R. Govindan, BGP Route Flap Damping,

RFC-2439, 1998.

[17] B. Chinoy, Dynamics of internet routing information, Proceedings of

ACM SIGCOMM (1993) 45–52.

[18] V. Paxson, End-to-end routing behavior in the internet, Proceedings of

ACM SIGCOMM (1996) 25–38.

[19] R. Govindan, A. Reddy, An analysis of inter-domain topology and

route stability, Proceedings of IEEE INFOCOM (1997) 850–857.

[20] C. Labovitz, G. Malan, F. Jahanian, Origins of internet routing

instability, Proceedings of IEEE INFOCOM (1999) 218–226.

[21] Internet Performance Measurement and Analysis (IPMA) (2000),

URL: http://www.merit.edu/ipma/.

[22] WorldCom MAE Information Site (2000), URL: http://www.mae.net/.

[23] Multithreaded Routing Toolkit (MRT) (2000), URL: http://www.

mrtd.net/.

[24] Routing Problems Archive, URL: http://www.merit.edu/mail.

archives/html/routing-problems/.


http://www.ripe.net/

http://www.apnic.net/

http://www.apnic.net/

http://www.merit.edu/ipma/

http://www.mae.net/

http://www.mrtd.net/

http://www.mrtd.net/

http://www.merit.edu/mail.archives/html/routing-problems/

http://www.merit.edu/mail.archives/html/routing-problems/

Characterizing and reducing route oscillations in the Internet

Documents

Transcript of Characterizing and reducing route oscillations in the Internet