Amogh Dhamdhere
Provider and Peer Selection in the Evolving Internet
Ecosystem
Committee:Dr. Constantine Dovrolis (advisor)Dr. Mostafa AmmarDr. Nick FeamsterDr. Ellen ZeguraDr. Walter Willinger (AT&T Labs-Research)
The Internet Ecosystem
27,000 autonomous networks independently operated and managed
The “Internet Ecosystem” Different types of networks Interact with each other and with “environment”
Network interactions Localized, in the form of interdomain links Competitive (customer-provider), symbiotic (peering)
Distributed optimizations by each network Select providers and peers to optimize utility function
The Internet ecosystem evolves04/19/23 2
High Level Questions
How does the Internet ecosystem evolve? What is the Internet heading towards?
Topology Economics Performance
How do the strategies used by networks affect their utility (profits/costs/performance)?
How do these individual strategies affect the global Internet?
04/19/23 3
Previous Work
Static graph properties No focus on how the graph
evolves
“Descriptive” modeling Match graph properties
e.g. degree distribution Homogeneity
Nodes and links all the same
Game theoretic, computational Restrictive assumptions
Dynamics of the evolving graph Birth/death Rewiring
“Bottom-up” Model the actions of
individual networks Heterogeneity
Networks with different incentives
Semantics of interdomain links
04/19/23 4
Our Approach
“Measure – Model – Predict”
04/19/23 5
Measure the evolution of the Internet EcosystemTopological, but focus on business types and rewiring
Model strategies and incentives of different network types
Predict the effects of provider and peer selection strategies
Outline
Measuring the Evolution of the Internet Ecosystem
The Core of the Internet: Provider and Peer Selection for Transit Providers
The Edge of the Internet: ISP and Egress Path Selection for Stub Networks
ISP Profitability and Network Neutrality
04/19/23 6
[IMC ’08]
[to be submitted]
[Netecon ’08]
[Infocom ‘06]
Motivation
How did the Internet ecosystem evolve during the last decade?
Is growth more important than rewiring? Is the population of transit providers increasing or
decreasing? Diversification or consolidation of transit
market? Given that the Internet grows in size, does the
average path length also increase? Where is the Internet heading?
04/19/23 7
Approach Focus on Autonomous Systems (ASes)
As opposed to networks without AS numbers Start from BGP routes from RouteViews and RIPE
monitors during 1997-2007 Focus on primary links
Classify ASes based on their business function Enterprise ASes, small transit providers, large transit
providers, access providers, content providers
Classify inter-AS relations as “transit” and “peering” Transit link – Customer pays provider Peering link – No money exchanged
04/19/23 8
Visibility Issue
Internet growth
Number of CP links and ASes showed initial exponential growth until mid-2001
Followed by linear growth until today Change in trajectory followed stock market crash in North
America in mid-200104/19/23 9
Path lengths stay constant
Number of ASes has grown from 5000 in 1998 to 27000 in 2007
Average path length remains almost constant at 4 hops
04/19/23 10
Rewiring more important than growth
Most new links due to internal rewiring and not birth (75%)
Most dead links are due to internal rewiring and not death (almost 90%)
04/19/23 11
Classification of ASes based on business function
Four AS types: Enterprise customers (EC) Small Transit Providers
(STP) Large Transit Providers
(LTP) Content, Access and
Hosting Providers (CAHP) Based on customer and
peer degrees Classification based on
decision-trees 80-85% accurate04/19/23 12
Evolution of AS types
LTPs: constant population (top-30 ASes in terms of customers) Slow growth of STPs (30% increase since 2001) EC and CAHP populations produce most growth
Since 2001: EC growth factor 2.5, CAHP growth factor 1.504/19/23 13
Multihoming by AS types
CAHPs have increased their multihoming degree significantly On the average, 8 providers for CAHPs today
Multihoming degree of ECs almost constant (average < 2) Densification of the Internet occurs at the core
04/19/23 14
Conjectures on the Evolution of peering
Peering by CAHPs has increased significantly CAHPs try to get close to sources/destinations of content
04/19/23 15
Conclusions Where is the Internet heading?
Initial exponential growth up to mid-2001, followed by linear growth phase
Average path length practically constant Rewiring more important than growth Need to classify ASes according to business type ECs contribute most of the overall growth Increasing multihoming degree for STPs, LTPs and
CAHPs Densification at the core
CAHPs are most active in terms of rewiring, while ECs are least active
1604/19/23
Outline
Measuring the Evolution of the Internet Ecosystem
The Core of the Internet: Provider and Peer Selection for Transit Providers
The Edge of the Internet: ISP and Egress Path Selection for Stub Networks
ISP Profitability and Network Neutrality
04/19/23 17
Modeling the Internet Ecosystem
From measurements: Significant rewiring activity Especially by transit providers
Networks rewire connectivity to optimize a certain objective function Distributed Localized spatially and temporally Rewiring by changing the set of providers and peers
What are the global, long-term effects of these distributed optimizations? Topology and traffic flow Economics Performance (path lengths)
04/19/23 18
The Feedback Loop
04/19/23 19
When does it converge?When no network has the incentive to change its connectivity – “steady-state”
InterdomainTM
Traffic flow
Interdomain topology
Per-ASprofit
Cost/priceparameters
Routing
Providerselectio
n
Peer selectio
n
Impact of provider/peer selection strategies
04/19/23 20
C CS
S S
C
C
P1
P2 P3
S
No peeringP2 and P3 peer
P1 open to peering with CPs
Our Approach
What is the outcome when networks use certain provider and peer selection strategies?
Model the feedback loop in the Internet ecosystem Real-world economics of transit, peering, operational costs Realistic routing policies Geographical constraints Provider and peer selection strategies
Computationally find a “steady-state” No network has further incentive to change connectivity
Measure properties of the steady-state Topology, traffic flow, economics
04/19/23 21
Network Types
Enterprise Customers (EC) Stub networks at the edge (e.g. Georgia Tech) Either sources or sinks
Small Transit Providers (STP) Provide Internet transit Mostly regional in presence (e.g. France Telecom)
Large Transit Providers (LTP) Transit providers with global presence (e.g. AT&T)
Content Providers (CP) Major sources of content (e.g. Google)
04/19/23 22
Provider and peer selection for STPs and LTPs
What would happen if..? The traffic matrix consists of mostly P2P traffic? P2P traffic benefits STPs, can make LTPs
unprofitable
LTPs peer with content providers? LTPs could harm STP profitability, at the expense of
longer end-to-end paths
Edge networks choose providers using path lengths?
LTPs would be profitable and end-to-end paths shorter
04/19/23 23
Provider and Peer Selection
Provider selection strategies Minimize monetary cost (PR) Minimize AS path lengths weighted by traffic (PF) Avoid selecting competitors as providers (SEL)
Peer selection strategies Peer only if necessary to maintain reachability (NC) Peer if traffic ratios are balanced (TR) Peer by cost-benefit analysis (CB)
Peer and provider selection are related
04/19/23 24
Provider and Peer Selection are Related
04/19/23 25
A A
C
C
B B
U
U
B B
U
U
A A
C
C
Peering by necessity
Level3-Cogent peering dispute
Restrictive peering
A A
CC
B B
?X
Economics, Routing and Traffic Matrix
Realistic transit, peering and operational costs Transit prices based on data from Norton Economies of scale
BGP-like routing policies No-valley, prefer customer, prefer peer routing policy
Traffic matrix Heavy-tailed content popularity and consumption by sinks Predominantly client-server: Traffic from CPs to ECs Predominantly peer-to-peer: Traffic between ECs
04/19/23 26
Algorithm for network actions
Networks perform their actions sequentially Can observe the actions of previous networks
And the effects of those actions Network actions in each move
Pick set of preferred providers Attempt to convert provider links to peering links “due to
necessity” Evaluate each existing peering link Evaluate new peering links
Networks make at most one change to their set of peers in a single move
04/19/23 27
Solving the Model
Determine the outcome as each network selects providers and peers according to its strategy
Too complex to solve analytically: Solve computationally
Typical computation Proceeds iteratively, networks act in a predefined
sequence Pick next node n to “play” its possible moves Compute routing, traffic flow, AS fitness Repeat until no player has incentive to move “steady-state” or equilibrium
04/19/23 28
Properties of the steady-state
Is steady-state always reached? Yes, in most cases
Is steady-state unique? No, can depend on playing sequence Different steady-states have qualitatively similar
properties
Multiple runs with different playing sequence Average over different runs Confidence intervals are narrow
04/19/23 29
Canonical Model
Parameterization of the model that resembles real world
Traffic matrix is predominantly client-server (80%) Impact of streaming video, centralized file sharing
services 20% of ECs are content sources, 80% sinks Heavy tailed popularity of traffic sources Edge networks choose providers based on price 5 geographical regions STPs cheaper than LTPs
04/19/23 30
Model Validation
Reproduces almost constant average path length Activity frequency: How often do networks change
their connectivity? ECs less active than providers – Qualitatively similar to
measurement results
04/19/23 31
Results – Canonical Model
04/19/23 32
EC
LTP
S1
S2 CP
CP
CP
EC
Hierarchy of STPs
Traffic can bypass LTPs – LTPs unprofitable
STPs should not peer with CPsResist the temptation!
Results – Canonical Model
04/19/23 33
LTP
S1
S2
CP
CP
CP
EC EC
What-if: LTPs peer with CPs
Generate revenue from downstream traffic
Can harm STP fitness
Long paths
Deviation 1: P2P Traffic matrix
04/19/23 34
LTP
S1
S2
CP
CP
CP
EC EC
P2P traffic helps STPsSmaller traffic volume from CPs to Ecs
More EC-EC traffic => balanced traffic ratiosMore opportunities for STPs to peerPeering by “traffic ratios” makes sense
LTP
S1
S2
EC EC
S3
EC EC
Conclusions
A model that captures the feedback loop between topology, traffic and fitness in the Internet
Considers effects of Economics Geography Heterogeneity in network types
Predict the effects of provider and peer selection strategies Topology, traffic flow, economics, and performance
04/19/23 35
Outline
Measuring the Evolution of the Internet Ecosystem
The Core of the Internet: Peer and Provider Selection for Transit Providers
The Edge of the Internet: ISP and Egress Path Selection for Stub Networks
ISP Profitability and Network Neutrality
04/19/23 36
The Edge of the Internet
Sources and sinks of content Content Providers (CP): sources Enterprise Customers (EC): sinks
From measurements: ECs connect increasingly to STPs
Cost conscious ? CPs connect increasingly to LTPs
Performance ?
Increasing multihoming (about 60% of stubs) Redundancy, load balancing, cost effectiveness
How should stub networks choose their providers?
04/19/23 37
Major Questions How to select the set of
upstream ISPs ? Low monetary cost Short AS paths to major
destinations Path diversity to major traffic
destinations – robustness to network failures
How to allocate egress traffic to the set of selected ISPs ? Objective: Avoid congestion on
the upstream paths Also maintain low cost
04/19/23 38
ISP Selection Select k ISPs out of N
Let C be a subset of k ISPs out of N
Total cost of a selection of ISPs C: Weighted sum of monetary, path length and path diversity costs
Select combination C with minimum total cost Feasible to enumerate all combinations
04/19/23 39
Monetary and Path Length Cost
For set of ISPs C, what is the monetary and path length cost of routing egress flows?
Find the minimum cost mapping G* of flows to ISPs (Bin Packing) Flows = items ISPs = bins NP hard !
Use First Fit Decreasing (FFD) heuristic Mapping G* very close to optimal
Monetary and path length costs of C are calculated using the mapping G*
04/19/23 40
Path Diversity selection C gives K paths to
each destination d
K-shared link to d: link shared by all K paths to d If a K-shared link fails, destination
d is unreachable Minimize the number of K-
shared links Path diversity metric: The number
of k-shared links to destination d averaged over all destinations
Gives the best resiliency to single-link failures
04/19/23 41
Summary Algorithms for ISP selection
Choosing best set of upstream ISPs Objectives are minimum monetary cost, short AS paths
and high path diversity
ISP selection for monetary and performance constraints Formulated as a bin-packing problem Heuristic gives solution very close to optimal
ISP selection for path diversity Returns set of ISPs with best path diversity to the set of
major destinations04/19/23 42
Outline
Measuring the Evolution of the Internet Ecosystem
The Edge of the Internet: ISP and Egress Path Selection for Stub Networks
The Core of the Internet: Peer and Provider Selection for Transit Providers
ISP Profitability and Network Neutrality
04/19/23 43
The debate
Recent evolution trend: Large amounts of video and peer-to-peer traffic
Content providers (CP) generate the content Provide content and services “over the top” of the basic
connectivity provided by ISPs Profitable (think Google)
Access Providers (AP) deliver content to users Recent trend: Not profitable Commoditization of basic Internet access Want a share of the pie
Tension between AP and CPs: “Network neutrality”
04/19/23 44
A Technical View
Previous work Mostly non-technical Highly emotional debates in the press Legislation/policy aspects: Do we need network
neutrality legislation? But what about the underlying problem: Non-
profitability of Access Providers? Our approach: A quantitative look at AP
profitability Investigate reasons for non-profitability Evaluate strategies for remaining profitable
04/19/23 45
Modeling AP Profitability
Three AS types: AP, CP and transit provider (TP)
Focus on the AP AS links
customer-provider (customer pays provider)
peering (no payments) AP and CPs can transfer
traffic either through customer-provider or peering links
04/19/23 46
AP Profitability
Reasons why APs can be unprofitable AP users The impact of video traffic
AP strategies: Pricing Heavy hitter charging Heavy hitter blocking Non-neutral charging
AP strategies: Connection Caching CP content Peering selectively with CPs
04/19/23 47
Major Findings
Variability in AP users can cause large variability in costs
Video traffic: Increases costs for AP
AP strategies based on differential/non-neutral pricing may not succeed Have to account for user departure due to competition
AP strategies based on connection are promising Caching content from CPs Peering selectively with large CPs
04/19/23 48
Contributions of this Thesis
A measurement study of the evolution of the Internet ecosystem
Modeling the evolution of the Internet ecosystem “what-if” questions about possible evolution paths
Optimizations at the edge of the Internet Algorithms for provider selection and egress routing
A technical view of the network neutrality debate Strategies for ISP profitability
04/19/23 49
Future Directions
Measurements: Investigate the evolution of the connectivity for monitor ASes We can observe all links for such ASes Focus on transitions between peering and customer-
provider links
Measurements: What does the interdomain traffic matrix really look like? Can we use measurements from a large Tier-1 provider? Can we augment that data with information about the
interdomain topology?
04/19/23 50
Future Directions
What is the best strategy for different types of providers? Strategies for classes of providers Strategies for individual providers
Do the distributed optimizations by networks solve a centralized problem? E.g., minimizing path lengths
04/19/23 51
Other things I’ve been up to Router buffer sizing
“Buffer Sizing For Congested Internet Links” “Open Issues in Router Buffer Sizing”
Network troubleshooting “NetDiagnoser: Troubleshooting network disruptions
using end-to-end probes and routing data” Network monitoring
“Route monitoring from passive data plane measurements”
Measurement “Poisson vs. Periodic Path Probing” “Bootstrapping in Gnutella”
04/19/23 52
[Infocom ‘05]
[CCR ‘06]
[CoNext ‘07]
[In progress]
[IMC ‘05]
[PAM ‘04]
Thank You !
04/19/23 53
04/19/23
Issue-1: remove backup/transient links
Each snapshot of the Internet topology captures 3 months 40 snapshots – 10 years
Perform “majority filtering” to remove backup and transient links from topology For each snapshot, collect several “topology samples”
interspersed over a period of 3 weeks Consider an AS-path only if it appears in the majority of
the topology samples Otherwise, the AS-path includes links that were active for
less than 11 days (probably backup or transient links)
Snapshot
Samples
04/19/23 54
Issue-2: variable set of BGP monitors
Some observed link births may be links revealed due to increased monitor set Similarly for observed link deaths
We calculated error bounds for link births and deaths Relative error < 10% for CP links See paper for details
04/19/23 55
Issue-3: visibility of ASes, Customer-Provider (CP) and Peering (PP) links
Number of ASes and CP links is robust to number of monitors
But we cannot reliably estimate the number of PP links
04/19/23 56
Global Internet trends
04/19/23 57
Transit (CP) vs Peering (PP) relations
The fraction of peering links has been increasing steadily But remember: this is just a lower bound
At least 20% of inter-AS links are of PP type today
04/19/23 58
The Internet gets larger but not longer
Average path length remains almost constant at 4 hops Average multihoming degree of providers increases faster
than that of stubs Densification at core much more important than at edges
04/19/23 59
04/19/23 60
Regional distribution of AS types
Europe is catching up with North America w.r.t the population of ECs and LTPs
CAHPs have always been more in Europe More STPS in Europe since 200204/19/23 61
Evolution of Internet transit: the customer’s perspective
04/19/23 62
Customer activity by region
Initially most active customers were in North America After 2004-05, customers in Europe have been more active
Due to increased availability of providers? More competitive market?
04/19/23 63
How common is multihoming among AS species?
CAHPs have increased their multihoming degree significantly On the average, 8 providers for CAHPs today
Multihoming degree of ECs has been almost constant (average < 2) Densification of the Internet occurs at the core
04/19/23 64
Who prefers large vs small transit providers?
After 2004, ECs prefer STPs than LTPs Mainly driven by lower prices or regional constraints?
CAHPs connect to LTPs and STPs with same probability
04/19/23 65
Customer activity by region
Initially most active customers were in North America After 2004-05, customers in Europe have been more active
Due to increased availability of providers? More competitive market?
04/19/23 66
Evolution of Internet transit: the provider’s perspective
04/19/23 67
Attractiveness (repulsiveness) of transit providers
Attractiveness of provider X: fraction of new CP links that connect to X Repulsiveness, defined similarly
Both metrics some positive correlation with customer degree Preferential attachment and preferential detachment of rewired links
04/19/23 68
Evolution of attractors and repellers
A few providers (50-60) account for 50% of total attractiveness (attractors)
The total number of attractors and repellers increases The Internet is NOT heading towards oligopoly of few large players
LTPs dominate set of attractors and repellers CAHPs are increasingly present however04/19/23 69
Correlation of attractiveness and repulsiveness
Timeseries of attractiveness and repulsiveness for each provider
Calculate cross-correlation at different lags Most significant correlation values at lags 1,2 and 3
Attractiveness precedes repulsiveness by 3-9 months04/19/23 70
Evolution of Internet peering (conjectures)
04/19/23 71
Evolution of Internet Peering
ECs and STPs have low peering frequency Aggressive peering by CAHPs after 2003
Open peering policies to reduce transit costs
04/19/23 72
Which AS pairs like to peer?
Peering by CAHPs has increased significantly CAHPs try to get close to sources/destinations of content
Peering by LTPs has remained almost constant (or declined) “Restrictive” peering by LTPs
04/19/23 73
04/19/23
Conclusions Where is the Internet heading
towards?
Initial exponential growth up to mid-2001, followed by linear growth phase
Average path length practically constant Rewiring more important than growth Need to classify ASes according to business type ECs contribute most of the overall growth Increasing multihoming degree for STPs, LTPs and
CAHPs Densification at core
CAHPs are most active in terms of rewiring, while ECs are least active04/19/23 74
Conclusions Where does the Internet head
toward?
Positive correlations between attractiveness & repulsiveness of provider and its customer degree
Strong attractiveness precedes strong repulsiveness by period of 3-9 months
Number of attractors and repellers between shows increasing trend
The Internet market will soon be larger in Europe than in North America In terms of number of transit providers and CAHPs
Providers from Europe increasingly feature in the set of attractors and repellers
04/19/23 75
Multihoming Multihoming: Connection of
a stub network to multiple ISPs x% of stub networks are
multihomed Redundancy
primary/backup relationships Load Balancing
Distribute outgoing traffic among ISPs
Cost Effectiveness Lower cost ISP for bulk traffic,
higher cost ISP for performance-sensitive traffic
Performance Intelligent Route Control
04/19/23 76
ISP selection
Should consider both monetary cost and performance
Minimum monetary cost Estimate the cost that “would be” incurred if a set of
ISPs was selected Minimum AS path lengths
Longer paths: delays, interdomain routing failures Measure AS path length offline using Looking Glass
Servers Maximum Path diversity
AS-level paths to destinations should be as “different” as possible
04/19/23 77
Problem Definition Two phases Phase I – ISP Selection:
Select K upstream ISPs K depends on monetary and performance constraints “Static” operation Change only when major changes in the traffic
destinations or ISP pricing Phase II – Egress Path Selection
Allocate egress traffic to selected ISPs Avoid long term congestion and minimize cost “Semi-static” operation, performed every few hours or
days
04/19/23 78
Evaluation – Path Diversity
AS-level paths and traffic rates are input to simulator 9 ISPs, 250 destinations
Given K, find the selection C* with the minimum path diversity cost
For each selection C, find u(C) = total traffic lost due to the failure of each link in topology
Calculate Δu(C) = u(C) – u(C*) for each selection C
04/19/23 79
Single link failures: C* is the optimal selection
Egress Path Selection After Phase-I, S has K upstream ISPs Problem: How to map outgoing traffic to the ISPs M flows: KM mappings of flows to ISPs
Some mappings may cause congestion to flows ! Flows can be congested at access links or further upstream
Objective: Find the loss-free mapping with the minimum cost
Challenges: Upstream topology and capacities are unknown Iterative routing approaches required
Propose an iterative routing based on simulated annealing
04/19/23 80
Evaluation – Path Diversity
AS-level paths and traffic rates are input to simulator 9 ISPs, 250 destinations
Given K, find the selection C* with the minimum path diversity cost
For each selection C, find u(C) = total traffic lost due to the failure of each link in topology
Calculate Δu(C) = u(C) – u(C*) for each selection C
04/19/23 81
Single link failures: C* is the optimal selection
2,3 link failures: C* is close to the optimal selection
Provider and Peer Selection
Detailed model for provider and peer selection Complex real-world decisions
Provider selection objectives Monetary cost AS path lengths
Peer selection Minimize transit costs Maintain reachability
Constraints Only local knowledge Geographical constraints
04/19/23 82
Peering Federation
Traditional peering links: Not transitive Peering federation of A, B, C: Allows mutual transit
Longer chain of “free” traffic Incentives to join peering federation? What happens to tier-1 providers if smaller providers form
federations?
04/19/23 83
A A B B C C
Why can the AP be unprofitable?
Variability of users => high variability in the costs incurred by AP
Variability increases with the access speed
Increase in video traffic: higher transit payment by AP
04/19/23 84
Baseline model AP and CP connect to the TP as customers N users of AP, charged a flat rate R ($/month) Transit pricing: 95th percentile of traffic volume,
concave transit pricing functions 95th / mean = 2:1 for normal traffic, 4:1 for video1
More video means higher transit payment by AP AP users: Heavy tailed distribution of content
downloaded per month High variability in AP costs
04/19/23 85
1Norton’06: Internet Video: The Next Wave of Massive Disruption to the U.S. Peering Ecosystem
AP Strategies Charging strategies
AP charges “heavy hitters” according to volume downloaded
AP caps heavy hitters AP charges CP (non-
network neutral) Charging strategies are
disruptive AP cannot control
customer departure probability
04/19/23 86
AP Strategies
Charging heavy hitters download amount D,
threshold T, flat rate R c(D) = D*R/T AP’s profit is sensitive to
customer departure prob
Capping heavy hitters and non-neutral charging would not work for the same reason
04/19/23 87
AP Strategies
Connection Strategies AP caches content from CPs AP peers with CPs
Non-disruptive Caching can reduce transit costs of AP
But depends on the amount of content cacheable Selective peering with CPs can improve
profitability Peering cost depends on CP Cost/benefit analysis for each CP
CP with large network: low cost of peering
04/19/23 88
AP Strategies Connection Strategies
AP caches content from CPs AP peers with CPs
Non-disruptive Cost-benefit analysis for
peering Peering cost depends on CP
(easy/medium/hard) r = saving/cost (both
estimated) Peer if r > R
AP controls the factor R
04/19/23 89
AP Strategies
Charging heavy hitters download amount D,
threshold T, flat rate R c(D) = D*R/T AP’s profit is sensitive to
customer departure prob Non-neutral charging
Customer departure prob “How discriminatory is
my AP?” AP’s profit is sensitive to
customer departure prob
04/19/23 90
Why Study Internet Evolution?
“Bottom-up” models (more later) Understand how local actions lead to emerging
properties
Performance of protocols over time “How would BGP perform 10 years from now?”
Clean slate vs. evolutionary design After initial design, both must evolve ! Understanding evolution of the current Internet can help
design
04/19/23 91
Top Related