HISTORY OF 100G AND INTERNET2Chris Robb, Indiana University/Internet2

• Introduction• Technical Background• History of Internet2’s 100G Deployment• Current Progress• Lessons Learned

Topics

• Why, I’m Chris Robb!• Director of Engineering and Operations for Internet2• But I work at IU. It’s all horribly confusing. • Started in the trenches of UITS computer lab support• Joined campus networking division as a programmer in 1998• GlobalNOC engineer in 2001. First assignment: Hawaii!• Managed the 2006 Internet2 transition to Level3• Subcontracted to Internet2 in 2008• My day consists of video conferencing, lunch, then more video

conferencing- usually in that order• I also occasionally haunt the halls of the CIB

Who’s this guy?

• “Internet2 is an advanced networking consortium led by the research and education community”

• Formed in 1996• More than just a network

– Middleware– Net+– Security– Research– Measurement

• Based in Ann Arbor with offices in Washington DC, New York and (soon) California

What’s Internet2?

Technical Background

National Backbone Organization

• Fiber• Optical Equipment• Routers and Switches• Network Operations Center

Dark Fiber• Someone has to lay fiber in the ground• Commonly laid along railway lines and interstate roads• A lot of investment in the late 90s• Multiple conduits installed

– Ability to blow fiber later• Multi-strand cable (144ct is common, per conduit)• Hut spacing defined at installation time• Huge investment

– Nationally cost prohibitive– Regionally expensive, but doable

• Sometimes not very well documented

Cable conduit is laid

Fiber is blown through empty conduit

Photos courtesy of Steve Wallace

Access to fiber via handholes

Fiber is terminated in huts to house optical amplification equipment

Who can buy fiber?• Depends on when you’re asking, who you’re asking, and who you are• Early 2000s: plentiful supply of fiber (and companies that are no

longer around)• Now: not so much

– Consolidation– Not a lot of investment in fiber (exception: high frequency trading)

• Mostly carrier swaps and internal use• Sold as an Indefeasible Right to Use (IRU), typically on a 20 year term• Expensive!

• Activate Google Earth

• What do you put on the fiber once it’s laid? – Too expensive to just put a single signal on each fiber

• Client interfaces can only go so far– 10GBase-SR – up to 300m (depending on fiber type)– 10Gbase-LR – 1310nm – 10km– 10GBase-ER – 1550nm – 40km– 10GBase-ZR – 1550nm – 80km

• Put a switch every 80km?– You saw those huts! Too little space.– Who wants to manage that many devices? Too complex.– There are only so many coal plants in the US. Too much power.

OK, we have some fiber….

• Multiplex!• Convert signals from client equipment (routers & switches) to specific

wavelengths of light (ITU grid)• Combine the different wavelengths onto a common fiber on one end,

and separate them on the other end• Uh oh! Light doesn’t travel far enough!

– Amplify!– Periodically shoot the signal through erbium doped fiber spools that

amplify the signal• The noise! The noise! Amplification also amplifies noise.

• Periodically regenerate the signal• Break down the analog signal back into its digital form, then send it

out again as a newly cleaned up analog signal

DWDM Optical Equipment

15

DWDM History• Early WDM (late 80s)

–Two widely separated wavelengths (1310, 1550nm)

• “Second generation” WDM (early 90s)–Two to eight channels in 1550 nm window–400+ GHz spacing

• DWDM systems (mid 90s)–16 to 40 channels in 1550 nm window–100 to 200 GHz spacing

• Next generation DWDM systems–64 to 160 channels in 1550 nm window–50 and 25 GHz spacing

ITU Wavelength Grid

• ITU-T l grid is based on 191.7 THz + 100 GHz• It is a standard for laser in DWDM systems

l1530.33 nm 1553.86 nm0.80 nm

195.9 THz 193.0 THz100 GHz

Freq (THz) ITU Ch Wave (nm) 15201/252 15216 15800 15540 15454192.90 29 1554.13 x x x x x192.85 1554.54192.80 28 1554.94 x x x x x192.75 1555.34192.70 27 1555.75 x x x x x192.65 1556.15192.60 26 1556.55 x x x x x

• Fiber• Erbium Doped Fiber Amplifier (EDFA) (“Amp”)• Optical Add/Drop Multiplexor (“OADM”)• Reconfigurable Optical Add/Drop Multiplexor (“ROADM”)• Tributary (“Trib”) card• And a few more concepts

– Directionless– Colorless

• To the whiteboard!!

Modern Optical Components

Optical Multiplexer

Optical De-multiplexer

Optical Add/Drop Multiplexer(OADM)

Transponder

DWDM Componentsl1

l2

l3

l1

l2

l3

850/1310 15xx

l1

l2

l3

l1...n

l1...n

Optical Amplifier(EDFA)

Optical AttenuatorVariable Optical Attenuator

Dispersion Compensator (DCM / DCU)

More DWDM Components

Optical Amplifier

Optical Add/Drop

Transmission Effects• Attenuation:

– Reduces power level with distance

• Dispersion and nonlinear effects: – Erodes clarity with distance and speed

• Noise and Jitter:Leading to a blurred image

• How do you get millions of people connected to this capacity?• Routers direct your data transfers around the country• Generally large boxes at the national level ($500K-$1M each)

– Higher interface speeds (10Gbps+)– Not a lot of edge features (traffic shaping, policing, etc.)

National Routed Network

Internet2 Network Topology Evolution

• 1998-2003– Cisco GSR routers– Qwest OC-12/OC-48 backbone– Average Connector Speed: 450-600Mbps

• 2003-2007– Juniper T640 routers– Qwest OC-192 backbone– Average Connector Speed: 1-4Gbps

• 2007-2009– Juniper T640/T1600 routers– Level(3) OC-192/10GigE backbone– Average Connector Speed: 8.3Gbps

• 2009-2011– Juniper MX960 Routers– Level3 Multiple 10GigE Backbone– Average Connectors Speed: 10+ Gbps

• 2011 and beyond– Juniper T-1600 Routers– Owned Infrastructure with 100GigE Backbone

Potential Edge Bandwidth Growth

4/1/

99

4/1/

00

4/1/

01

4/1/

02

4/1/

03

4/1/

04

4/1/

05

4/1/

06

4/1/

07

4/1/

08

4/1/

09

4/1/

10

4/1/

110

2

4

6

8

10

12 11.210.3

Measured Perabytes into Internet2 per Month

1997 1998 2002 2008 2010 2011+0

10

20

30

40

50

60

70

80

90

100

0.6 2.410

20

30

100

Internet2 Network Backbone Speeds (in Gbps)

Internet2’s McLean, VA POP

Internet2’s McLean, VA POP

History of 100G Deployment

A Community Defines Its Future

• In June 2009, Internet2 and the R&E community author the “Internet2 Architectural Directions” Document….

• Multiples of 10GigE will be the primary transport to Regional and State Networks over the next 3-5 years.– 10G cost low compared to the cost of 40 or 100G– Multiple large sub-10G flows the norm

• Internet2 Network access will be divorced from physical interface speeds and available for apportionment across the network– Flexibility for connectors an important success factor.

Architectural Principles

• Native 100GigE at the optical layer is an important technology to adopt today– Take advantage of current opportunities to lay the

groundwork for future expansion.

• Collapsing Layer2 and Layer3 services onto a single delivery platform is an important step toward the hybridization of the network – Reduce overall operating expenses to the Connectors– Candidate technologies include MPLS L2 VPNs, Layer2

Ethernet VLANs and Virtual Private LAN Service (VPLS).

Architectural Principles

• The Internet2 IP and Layer2 Networks need a migration path to 40G and 100G in the next few years – Backbone must be able to efficiently handle multiple

simultaneous 7-10 G flows and individual flows >10Gbps

• The Internet2 Network emphasis should be on additional services and technologies that will drive transport bandwidth requirements – The use case for the network drives the technology of

the network.

Architectural Principles

• Internet2 will coordinate with the Regional and State Network partners to determine the most optimal node quantity and locations – Offer a flexible partnership with the connectors.– Create more options for connections.

• As mission-critical applications become more integral to the Regional cost-recovery model, the Internet2 Network must focus on enhanced redundancy where needed – Many recent services and uses of the network require

increasingly reliable/redundant/resilient connectivity

Architectural Principles

• The Internet2 Network will continue to be instrumented and operated in a transparent fashion that supports the end-to-end model – The more information that is available about

the network the better everyone understands the need for and requirements of the network.

Why is 100G So Important?

• Edge speeds are outpacing backbone speeds– mid 2009 potential edge bandwidth was 17x the edge

bandwidth of 2001

• Traffic is steadily growing• Large flows are becoming increasingly

commonplace• 100Gbps will scale and become economically

cost effective• Drive to innovate!

History of 100G and Internet2

• June 2009: Internet2 staff began polling the community for set of technical principles to guide staff efforts over the next several years– Resulted in the “Internet2 Architectural Directions”

document that specifically called out 100Gbps networking as a strategic direction

– AOAC approval of document on October 5th, 2009• August 2009 Internet2 released an RFI to the

industry that sought industry feedback on a 100Gbps optical platform

Funding History

• In late summer of 2009, the NTIA released a Notice of Funding Availability (NOFA) for broadband funding– Broadband Technologies Opportunities Program

(BTOP)– $7.2B in funding

• Internet2 decided not to respond to the first round of funding so as not to compete with the regional networks that were applying for funds

Funding

• Second round of BTOP funding in early 2010• Internet2 submits a joint proposal with NLR, IU and

the Northern Tier Networking Consortium (NTNC)– Obtain a partial national footprint and equip it with an

optical network– Leverage the existing NLR and Internet2 optical paths– Upgrade the NTNC network between Chicago and Seattle– Upgrade all Internet2 routers to be 100G capable– Upgrade the TR-CPS routers to newer hardware– Create a “low-latency” layer of small routers at 27 sites

Award and Change of Plans

• Internet2/NLR/NTNC were selected as the only “national middle mile” network in the BTOP program in late spring 2010

• During the due diligence phase, NLR opted to step out of the partnership– Network topology was refactored to a national

footprint to cover those sections of the country that had been covered by NLR’s in-kind contribution

• Project officially started in July, 2010

Preliminary Optical RFI

• February 2010: meetings were scheduled with several companies to review their offerings– Evaluation was based on a number of different factors:

• Optical properties• Control plane development • Physical properties• Road map for product availability and others

– From this review it was determined that there were optical vendors with viable options for building the U.S. UCAN network

– Accordingly they were given the option to respond to the RFP for optical equipment in September 2010

RFP Issuance

• Based on this research potential vendors were identified: – Cisco– Infinera– Ciena

• Internet2 released the U.S. UCAN Optical Network Request for Proposal (RFP) on September 13, 2010, in collaboration with Indiana University

• Vendor responses were due on September 22– The response due date was extended three times:

• on September 17 Internet2 granted an extension to September 29 • on September 23 Internet2 granted an extension to October 5th • on October 5 Internet2 granted an extension to October 9th

Review Team

• At the September NTAC meeting, Internet2 solicited volunteers from the community to review the RFP responses

• A team of community members, Internet2 staff and IU NOC staff was assembled to evaluate the responses

Team Outcome

• All three responses were outstanding and each vendor included compelling and creative support for this community effort. – Infinera will provide an upgrade path for the Northern Tier

Networking Consortium on their next generation platform– Ciena will provide the national footprint on its Activeflex

(formerly OME) 6500 platform– Cisco is being engaged on a variety of fronts for parallel efforts

• Recommendation presented to the AOAC on November 10th

– AOAC endorsement and forward to Internet2 senior management for final approval

Ciena Platform

• 100Gbps capable 80-channel DWDM system– 100G cards shipping today

• ROADM-based solution at most or all add/drop facilities

• Directionless capability in metro areas• Non-Dispersion-Shifted approach provides

economical approach that reduces CAPEX• Compact, scalable footprint that adapts to the

changing needs of our community

Current Progress

• Network Infrastructure– Dark Fiber on a national footprint– Wave Capacity

• Optical equipment to light new fiber with 100Gbps-capable equipment• Upgrade of Northern Tier Networking Consortium Infinera equipment

to be 100G-capable– 100Gbps IP Backbone

• 100G and 10G routers for R&E network• 10G routers for TR/CPS

• Enhanced Services– Increased connectivity to Commercial Exchange Points– Regional interconnects

• Research Opportunity– Stronger collaboration– Enhanced capability

Upgraded Internet2 Infrastructure benefits

48 – 04/09/2023, © 2009 Internet2

Internet2 Optical Network Topology

Phase 1 Optical Build - COMPLETE

• All fiber acquired and accepted• All Optical Equipment Installed• BER Testing complete• System Commissioning

– EMS system installed and populated with nodes– Internet2 NOC database population nearly complete

• First 100GigE IP circuit between New York and Washington DC live and integrated into the Internet2 IP Network

• First Trans-Continental 100G configured and passing pings for Internet2 Fall Member Meeting in early October

Phase 1 Progress – 100% Complete

Phase 2 – Complete

52 – 04/09/2023, © 2009 Internet2

Phase 3 – January 2012 – July 2013

53 – 04/09/2023, © 2009 Internet2

Current Status

• Optical is great, but what about the routers?• Well, that’s a tricky story. • We’re at a crossroads• Routers are expensive, big, power hungry monsters• Did I mention expensive?• We want them everywhere• We want them cheap• We want them small• We want control• We want them powerful

• Is there a white knight?

Will someone please think of the routers?

Openflow?

• Faster– 2002: One of the first 10G IP national backbones deployed on Juniper T640

platform– Mid-2009: Architectural Directions document cites need for 100G IP backbone– Mid-2010: $96M Broadband Technologies Opportunities Program (BTOP)

National Middle Mile Award– Late-2011: First national 100GigE circuit lit between New York and Sunnyvale

• Smarter– 2005: Hybrid Optical Packet Infrastructure (HOPI) network deployed in first

national trial of reconfigurable network– 2007: Interoperable On-demand Network (ION) deployed at 21 locations

around the US– 2011: DYNES campus deployment project begins pushing dynamic deployment

to the edge– Mid-2011: Open Networking Foundation formed by a coalition of research and

industry members– Late-2011: 5-node NDDI deployment and development of OSE software

History of implementation

57 – 04/09/2023, © 2011 Internet2

• Demand from Internet2 Members to Innovate• Equipment Refresh Cycle• Available Funding

Why now?

58 – 04/09/2023, © 2011 Internet2

• This is a very high-level restructure of the Internet2 network, so it’s important that stakeholders at all levels participate in the discussion– Researchers– Campuses– Regionals– International Peers– Funding Agencies– Governance Councils– Technical Bodies

• Internet2 will be aggressive in its deployment of its Innovation Platform in order to allow its members to capitalize on the groundswell of support for high-speed software defined networking

Engagement Plan

59 – 04/09/2023, © 2011 Internet2

• Goal: Integrate SDN functionality into the Internet2 Network– Preferred Goal: platform integration between two sets of services to

provide initial network effect and migration path• A few different models to consider• Nothing is set in stone, but we do have to move aggressively

Implementation possibilities

60 – 04/09/2023, © 2011 Internet2

Separate Infrastructure: What we do today

61 – 04/09/2023, © 2011 Internet2

Combined Infrastructure: What we might do tomorrow

62 – 04/09/2023, © 2011 Internet2

Upgraded Layer2 Network

• Pre-project– Know your goals– Know your limitations, then ignore them– Community Input

• Negotiations– Have a good lawyer– Understand what you’re buying– Budget a lot of time

• Remember: this is for 20+ years

• Implementation– Documentation is critical– Inventory management needs to be worked out ahead of time– It isn’t plug and play– Budget a lot for space and power

Lessons Learned

QUESTIONS?