Terabit BackbonesA Reality Check

Vijay Gill <vgill@mfnx.net>

Agenda

Current State of the Internet Side detour through

VPNs DiffServ/QoS/CoS The Converged Core (hype machine

that goes to 11)

State Of the Internet Address

1. Amount of state at any level should be constrained and must not exceed Moore’s Law for economically viable solutions.

2. Ideally – growth of state should trail Moore’s Law

We’re in trouble“If you’re not scared, you don’t understand” – Mike O’Dell

Reality Based Internet Economics

Growth of State

Recent trends show high growth in Internet state (routes, prefixes etc.)

Isolate this growth as a predictor of future growth

Compare growth to Moore’s law

50000

60000

70000

80000

90000

100000

110000

120000

130000

140000

150000

01/99 04/99 07/99 10/99 01/00 04/00 07/00 10/00 01/01 04/01

Pre

fixe

sGlobal routing table vs Moore's law

Global prefixesMoore's law

Source: Tony Li (Procket Networks)

50000

60000

70000

80000

90000

100000

110000

120000

130000

140000

150000

160000

01/99 04/99 07/99 10/99 01/00 04/00 07/00 10/00 01/01 04/01

Pre

fixe

sGlobal routing table vs Moore's law

Global prefixesMoore's law

Double growth

Source: Tony Li (Procket Networks)

The Very Bad News

Growth rate is Increasing Hyper-exponential growth

Will eventually outgrow Moore’s law Moore’s law may fail

50000

60000

70000

80000

90000

100000

110000

01/99 04/99 07/99 10/99 01/00 04/00 07/00 10/00 01/01 04/01

Pre

fixe

s (L

og

Sca

le)

Global routing table size since 1999

Prefix growthExponential growth

Source: Tony Li (Procket Networks)

The Real Problems

If we don’t fix these, the other problems won’t matter Hyper-exponential growth will exceed Moore’s

law Safety margins are at risk We need concerted effort on a new routing

architecture Multi-homing must not require global prefixes Example: IPv6 plus EIDs

Nov 1999 - 16,000 individual addresses Dec 2000 - 12,100 individual addresses

Increase in the average prefix length from /18.03 to /18.44. Dense peering (Rise of Exchange Points) and Multi-homing

BGP Advertisement Span

Source: Geoff Houston

State Now

# of Paths vs. # of Prefixes Large amounts of peering CPU to crunch RIB to populate FIB More state requires more CPU time

Leads to Delayed Convergence BGP – TCP rate limited, just adding pure

CPU isn’t the entire answer Issue is with propagating state around

Convergence Times

030 35 50

100

195

360

550

810

1114

0

200

400

600

800

1,000

1,200

0 20000 40000 60000 80000 100000 120000 140000 160000 180000 200000

Routes

Seco

nd

s

Problem With State

Issues with interactions of increased state, CPU, and message processing

Run to completion processing <-> missed hellos IGP meltdowns

Time diameter exceeds hold down Pegged CPU on primary causes slave to initiate

takeover Decoupled Hello processing from Routing

Process

VoIP? What VoIP?

IGPs Converge on average converge an order of magnitude faster than BGP

Leads to temporary black holing Router reboots (like that ever happens) IGP converges away, BGP teardown

Router comes back up IGP converges and places router in forwarding

path BGP is still converging

Packets check in, but don’t check out

VPNs - Operational Reality Check Vendors can barely keep one routing table

straight Customer Enragement Feature, IBGP withdraw bugs Into this mess, we’re going to throw in another

couple of hundred routing tables like some VPN proposals?

Potential for several thousand internal customer prefixes inside our Edge routers Revenge of RIP

Provider Provisioned VPNs – Just Say No.

What Is Going to Work

Some people will optimize for high touch edges – Provider provisioned VPNs etc. But if they are talking with the rest of

the world, welcome to the new reality – It sucks.

For the Rest….

“Already, data dominates voice traffic on our networks”

-Fred Douglis, ATT Labs

These exhibits were originally published in Peter Ewens, Simon Landless, and Stagg Newman, "Showing some backbone," The McKinsey Quarterly, 2001 Number 1, and can be found on the publication's Web site, www.mckinseyquarterly.com. Used by permission.

What to optimize for

Optimize for IP Parallel backbones

Some ISPs already have to do this based on volume of traffic for IP alone

Do not cross the streams Voice traffic has well known properties

Utilize them

Optical network – Utilize DWDM and OXCs to virtualize the fiber

Solution

Internet (IP) Internet (IP)

VPN VPN

Voice/Video

CES

Voice/Video

CESMulti Service Optical

Transport Fabric

NEWS FLASH

Simple & Stupid Trumps Complex & Smart Every Time

Networks Powered by PowerPoint ™ Stuff looks good on slides, then we try and

hire people to implement and operate it Operational Reality Beats PowerPoint

every time

The Converged Core ™

For the fortunate few Utilize OXCs + DWDM to impose

arbitrary topologies onto fiber For the rest trying to run IP over

Voice… Nice knowing you…. Voice - Use SONET as normal, it’s not

growing very fast, so don’t mess with it WCOM, T

Network Design Principle

The main problem is SCALING Everything else is a secondary If we can scale, we’re doing something

right State Mitigation

Partition State What you don’t know, can’t hurt you

Common Backbone

Application Unaware Rapid innovation

Clean separation between transport, service, and application

Allows new applications to be constructed without modification to the transport fabric.

Less Complex (overall)

Why A Common Backbone?

Spend once, use many Easier capacity planning and

implementation Elastic Demand

Increase of N on edge necessitates 3-4 N core growth

Flexibility in upgrading bandwidth allows you to drop pricing faster than rivals

These exhibits were originally published in Peter Ewens, Simon Landless, and Stagg Newman, "Showing some backbone," The McKinsey Quarterly, 2001 Number 1, and can be found on the publication's Web site, www.mckinseyquarterly.com. Used by permission.

By carrying more traffic, a carrier can lower costs by up to 64%

Source: KPCB

Historical and forecast market price and unit cost of Transatlantic STM-1 circuit (on 25 year IRU lease)

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

18,000

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Pri

ce p

er S

TM

-1 (

$m)

PRICE

COST

Bandwidth Blender - Set on Frappe

Problem

We keep hearing the phrase ‘bandwidth glut’ So are we experiencing a glut or not?

No matter how much terabits of core bandwidth gets turned up….

Capacity Constraints are at the edges Go drop 2-4 racks in colocation facilities Q in QoS stands for Quantity, not Quality We don’t need to boil the oceans, all we

want is a poached fish

How To Build A Stupid Backbone Optical backbones cannot scale at the

STS-1 level High speed backbone reduces complexity

and increases manageability…. Impose a Hierarchy

Optical Backbone provides High-speed provisioning/management: OC-192/48

Sub-rate clouds multiplex lower speed traffic onto core lightpaths

Regional-Core Network Infrastructure

Core OXC

Multi-Service Platform

Client equipment

Core network

Metro SubNetwork Metro SubNetwork

Metro SubNetwork

Requirements

Support multiple services Voice, VPN, Internet, Private Line

Improving service availability with stable approaches where possible

Use MPLS if your SONET ring builds are taking too long (anyone still building SONET rings for data?)

If you have to use MPLS….

LSPs re-instantiated as p2p links in IGP e.g. ATL to DEN LSP looks like p2p link

with metric XYZ Helps obviate BGP blackholing

issues

Stabilize The Edge

Stabilize The Core

Global instability propagated via BGP Fate sharing with the global Internet

All decisions are made at the edge where the traffic comes in

Rethink functionality of BGP in the core

LSP Distribution

LDP alongside RSVP Routers on edge of RSVP domain do fan-out Multiple Levels of Label Stacking Backup LSPs

Primary and Backup in RSVP Core Speed convergence Removes hold down issues (signaling too fast in a

bouncing network) Protect path should be separate from working

There are other ways, including RSVP E2E

Implementation

IP + Optical Virtual Fiber Mesh Protection Overlay We already know where the big traffic will

be NFL Cities, London, Paris, Frankfurt, Amsterdam DWDM + Routers

IP + Optical

Fiber

DWDM / 3R

IP / Routers

Optical Switching

• Virtual Fiber • Embed Arbitrary fiber

topology onto physical fiber.• Mesh restoration. • Private Line• Increased Velocity of service

provisioning• Higher cost, added complexity

Edge

Core

Optical Switch

DWDM Terminal

Backbone Fiber

Metro Collectors

IP + Optical Network

Big Flow

Big Flow

Out of port capacity, switching speeds on routers? Bypass intermediate hops

Dual Network Layers

Optical Core (DWDM Fronted by OXC) Fast Lightpath provisioning Remember - Routers are very expensive OEO devices

Metro/Sub-rate Collectors Multiservice Platforms, Edge Optical Switches Groom into lightpaths or dense fiber. Demux in the PoP (light or fiber)

Eat Own Dog Food Utilize customer private line provisioning internally to

run IP network.

Questions

Vijay “Route around the congestion, we must” Gill

Nota Bene – This is not a statement of direction for MFN!

Many thanks to Tellium (Bala Rajagopalan and Krishna Bala) for providing icons at short notice!