The Evolution of the U.S. Internet Peering Ecosystem

The Evolution of the U.S. Internet Peering Ecosystem

William B. NortonCo-Founder and Chief Technical LiaisonEquinix, Inc.

Internet Researcher90% with Internet Companies EngineersEQIX: Massive Carrier Neutral ColocationLots of Blinking lightsObserve: documentation on HW&ProtocolsLack of Operations documentsResearch How does Peering work?White paper process..

Community Operations ResearchGround Truth w/dozens of expertsWrite White Paper v0.1Walk community through WP for commentsRevise White Paper into new versionPresent White Paper at conferencesSolicit comments over lunches and dinnersWhite papers so far

Internet Operations White PapersInterconnection Strategies for ISPsInternet Service Providers and PeeringA Business Case for PeeringThe Art of Peering: The Peering PlaybookThe Peering Simulation GameDo ATM-based Internet Exchanges Make Sense Anymore? Freely available. Send e-mail to [email protected] white Paper: The Evolution of the U.S. Peering Ecosystem, Agenda

AgendaDefinitions to level setPeering, Transit, ISPsSynch Point here.Introduction to notion:Peering EcosystemInternet=Many Internet Regions each withTier 1 ISPs, Tier 2 ISPs, Content ProvidersEvolution of the U.S. Peering Ecosystem3 major evolutions

Def: Internet Service ProviderDef: Internet Service Provider is an entity that sells access to the entire Internet. This service is often called Internet TransitDef: Transit is a business relationship whereby one entity sells access to the entire Internet.

Definition of Transit

1) ISP A buys Transit ServiceUpstream Transit ProviderUpstream Transit ProviderUpstream Transit ProviderUpstream Transit ProvidersDef: Transit is the business relationship whereby one ISP announces (sells) reachability to the *entire* Internet to a customer.$$$ ? Typically usage-basedPricing 04: $18-$200/MbpsVolume based on 95th Percentile measureTransit is Simple, Convenient: Upstream handles the delivery of packets to the Internet by some meansUsage: Im purchasing transit from Level 3.3) Traffic FlowsINTERNET

NETWORKS2) ReachabilityAnnouncement

Cost of Transit Traffic Exchange2001 Pricing Sampling Range: $100-$1200/MbpsMy Financial Models used $388/MbpsSource: 2004 Survey Avg. Range: $20-$400/Mbps 95th PercentileWhy Peer?1/15/04 field spot quotes: GX: $65/Mbps, HE: $25/Mbps Cogent $20/Mbps UU: $45/Mbps, etc. varied commits 0-1000Mbps

Scaling at the EdgeContent Heavy ISPAccess Heavy ISPUPSTREAMSVIGMBTStreaming Media56k384k1.5m$$$$$$Broadcast.com: 100,000 concurrent unicast streams15 million streaming hrs/mo, 1300 Live events/dayAccessAOL+ DSL: 1,000/dayRoadrunner Cable Modems: 1M subscribersNegotiate Peering with Upstream ISPsOr focus on peering with like-minded ISPs1 T1 consumer streaming 24/7 cost $388/mo in transitStreaming:BroadcastTelephonyVideoMultimediaInteractiveGamingSPOT EventsTier1 Equipment: $13B/00 -> $42B/04(Source: Infonetics)The Zone: +1M gaming user/mo!

Why Peer? Motivations for PeeringFinancial: Reduce load on expensive Transit serviceTraffic src/destMeasure vs IntuitUsage-based BillingEngineering: Lower latency

1st Stage of Peering:Top 10 destination ISP list

Transit$$$Transit$$$ISP AISP BTransit ISPTop 10, Def: Peering

Sample Top 10 Destination ListDef: Peering

Definition of PeeringDef: Peering is the business relationship whereby ISPs reciprocally announce reachability to each others transit customers.PeeringUSNetEastNetWestNetPeeringTransitRoutingTablesPeering is *not* a transitive relationshipPeering *does not* provide access to the entire InternetCost Peering vs. TransitUsage: I buy transit from UUNet and Peer with EastNet, WestNet.

The Cost of Peering1) Monthly Circuit Fee into IX2) Monthly IX Port Fees

When does Peering Make Sense Financially?

Generalization:ISP Peering Breakeven Analysis GraphsNumber of Mbps exchanged$/MbpsExchangedCost of TransitCost of Traffic Exchange in Peering RelationshipBreakeven Point (ISPs Indifferent between Peering and Transit traffic exchange)Prefer PeeringPeeringRisk

Synch PointYou should have a working lexicon at this pointA sense of the motivations for Peering & TransitAny questions so far?

The Internet Peering EcosystemPre-Crash : circa 2000

Peering EcosystemSystem of interconnected (Peering and Transit) playersEach group with homogenous peering motivations and observed behaviorsTier 1 ISPsTier 2 ISPsContent Providers

Def: Tier 1 ISPDefinition: A Tier 1 ISP is an ISP that has access to the entire Internet Region solely through peering relationships. (i.e. doesnt buy transit from anyone to see regional routes)Motivation: Doesnt need to peer with anyone else, soPeering only with other Tier 1 ISPs

Def: Tier 2 ISPDefinition: A Tier 2 ISPs is any ISP that has to purchase transit to reach destinations in an Internet RegionMotivation: Peer to decrease costs, improve performancePeering: with Tier 1s is preferable but hard to get, peering with Tier 2s as appropriateMust purchase Transit

Def: Content ProvidersDef: A Content Provider is an entity that produces content for delivery over the Internet but does not sell transit.Motivation: Best Customer Experience (engineered performance)Peering: Not typically done, prefer SLAs.Transit from ISPs

1998-2000ContentTier 2 ISPsTier 1 ISPsTransit$Thickness representsIn-Group Peering

The Evolution of the U.S. Peering Ecosystem

Act II: 2000-2003

Evolution of the U.S. Peering EcosystemFour key events leading to drastic disruption in the Peering Ecosystem:The 1999/2000 Telecom CollapseThe growth of the Used Equipment MarketThe Upstream Provider for the Cable Companies (@Home) went bankruptPeer-to-Peer file sharing systems (like Kazaa) grow in popularity, traffic grows exponentially between Access Providers (1.5MB MP3 700MB AVI files)Transit Prices Drop, Transport Prices Drop

Leads to 3 Major Evolutions

Evolution #1 Cable Companies are Peering40% of traffic Kazaa3-5 Gbps of transit traffic-> Gbps of peering potential !Significant becauseVolumeOpen PeeringKazaa Effect

U.S. Cable Companies Eyeballs

2000-2003 Evolution #1 Cable Companies PeeringContentTier 2 ISPsCableCosTier 1 ISPsPeering

Evolution #2 Network Savvy Large Scale Content Companies get into PeeringTo reduce transit costs To improve the end user experienceThey need to move out of bankrupt colo anywaySignificant becauseVolume of traffic is hugeContent Providers have Open PeeringLeading players pave the way

2000-2003 Evolution #2 Large Scale Network Savvy Content PeersContentTier 2 ISPsLSNSCTier 1 ISPs

Other Broadband PlayersSignificant, butNot Open Peers so disincentive to peerSo less volume and less disruptive

Evolution #3 Cable Companies Freely Peer with Content CompaniesContent literally on the Cable Company NetworkLowest possible latency from content to eyeballsInternet Gaming, Broadband Streaming, etc.

2000-2003 Evolution #1 Cable Companies PeeringContentTier 2 ISPsCableCosTier 1 ISPs

2000-2003 Evolution #2 Large Scale Network Savvy Content PeersContentTier 2 ISPsLSNSCTier 1 ISPs

2000-2003 Evolution #3 Cable Peers with Large Scale Network Savvy ContentContentTier 2 ISPsLSNSCTier 1 ISPsCableCos

International Dynamics Separate White PaperJPU.S.U.S. Tier 1 ISPs are Tier 2 ISPs in Japan Internet RegionJapan Tier 1 ISPs are Tier 2 ISPs in the U.S. Internet Region

Summary 2003 Peering Ecosystem EvolvingExcite@Home

NoPeeringOpenSelectiveRestrictiveLg.ContentTier2 ISPsCableCosRBOCsTier 1 ISPsForeign ISPs

The Evolution of the U.S. Internet Peering Ecosystem

Documents

Transcript of The Evolution of the U.S. Internet Peering Ecosystem