Dan Magorian

Director of Engineering &

Operations

What’s MAX Been Up to Lately?

Presentation to MAX membership

Spring 06 Member Meeting

MAXActivities

Production Net Activities

GIG-EFResearch

Technologies

Internet2 HOPI Test

Bed support

MAX Production Net and Research Activities

NGIX/East,AtlanticWave

peer points

DRAGONExperimental Networking

Disaster Recovery

Colocation

Research Activities

Layer 3 IP NetworkServices

Layer 1 Optical Services

Additional Funded

Research

Where we were a year ago

OADM

BALT

Qwest

OC48c POS

OC48c POS

OC48c POS

Abilene

M160 Router

M160 Router

Aggregation Switch

T640 Router

M40e Router

ARLG

DCGW DCNE

CLPK

NGIX

Northern Virginia

Washington D.C.

College Park

Baltimore

CLPKCLPK

CLPKCLPK

CLPKCLPK

MCLNMCLN

BALTBALT

MCLNMCLN

ASHBASHB ARLGARLG

DCGWDCGW DCNEDCNE

NLR

NGIX

T640T640T640

M40EM40E

Cogent

R&E Nets

Abilene

Qwest

Qwest

Where we are Spring 061. Original Zhone dwdm over Qwest fiber2. Movaz dwdm over State Md fiber3. HHMI dwdm over Abovenet fiber

4. (July) USM dwdm over State Md fiber

Ring1

Ring2

Ring3

Ring4

Has been/will be completed by July 06• Protected ring out of fiber CLPK – Level3 pop Mclean• Built out to Equinix public peering facility with help of HHMI (thanks!).• Redundancy/doubled capacity Qwest ISP via Equinix. Pioneered

low-cost “backup” 50mb rate vs 200mb with Qwest for Quilt folks.• After extended legal process, 400M Qwest ISP and I2 sold to HHS

via Qwest Federal Systems with MAX as subcontractor.• At Equinix, likely will transit IPv6 carrier peerings to Internet2, making

MAX primary east coast IPv6 exchange for them.• Brought up Cogent ISP through GWU. Anyone interested in $25/mb?• Good uptake on 2.5 & 10G lambdas to NLR: 4 leased, 6 in works• Phased out WhatsUp, deployed new Nagios-based open-source

monitoring and alerting system for bgp routes as well as link state. • Hosted I2 performance workshop, built portable 10G test pcs

available to members, deployed new noc and backup systems.• Hired 2 engineers, search for AtlanticWave engineer in progress.

Projects Underway FY06-07• 10G L3 backbone upgrade (more detail on following slide)• Baltimore dwdm ring. USM selected MRV, was ready to go, but

MDOT wanted in on project at 11th hour with additional sites and lambdas. Project expected to be online July. There will also be new 10G-capable Force10 switches at ring locations.

• More use of external lambdas, dropping costs, new vendor Ekinops (French) has cost-effective dwdm add-in gear.

• Atlantic Wave switch partnership with Ciena, will be used for upcoming GEANT peering (more detail on following slide)

• Advanced Exfo chromatic dispersion/polarization mode dispersion test gear just ordered, needed as we provision 10G waves over contributed unknown fiber. Will get additional test gear in 07 for sonet & ethernet analyzers. Loanable members.

• Continuing measurement and alerting infrastructure buildout for redundancy, alerting on optical, full info tracking database.

10G L3 backbone/Abilene backup upgrade

• Sun still intending to join MAX at 10G, but their grid computing project is delayed. Still expect to see 10G connectors in FY07.

• However, Abilene backup with another gigapop, either MAGPI or SOX, may happen sooner in 07. Will need 10G for this, also NLR wave or other connection.

• Need to move beyond current gige Abilene redundancy with CLPK and LVL3 routers as grow, to avoid having “eggs in same basket”.

• Hard part has been getting Juniper 10G T640 router interfaces at reasonable cost. Finally found one pair used.

• Finally convinced Juniper that need to offer 10G ints at high discount to all of higher ed to avoid missing out on 10G market. They will offer this to Quilt members at about 70% discount in 1-2 months..

• The way lambda costs have been dropping, only makes sense to provision 10G lambdas these days, and do 2.5G as subrate. FY05 10G transponder pair: $77k. FY06: $54k. Now, just paid $24k. Now as soon as we can get Juniper to follow suit…

Mid-Atlantic Crossroads

GEANT

Network Virginia

National/InternationalPeering Networks

Regional NetworkParticipants GIG-EF

ESnet

NGIX-East

DRENvBNS

NISN

NRENAbilene

MAXMAXRegional

Infrastructure

Qwest ISP

Institutional Connectors

Univ System Md

DRAGON

Cogent

Strong Carrier Partnerships

• With 7 MAX pop locations around region, participants can connect at one or more locations via dark fiber or lit services via aggressive low-cost carriers, as well as traditional telcos.

• 80% & growing of participant circuits ethernet.

• Strong partnerships with carriers (12 & growing!)• Qwest, Level 3, Fibergate, Starpower/RCN, Yipes,

Allied Telecom, AboveNet, Looking Glass Nets, Verizon, MCI, ATT, Comcast.

MAX and NGIX/E upstream network peerings

• Abilene (diverse: 2.5 gb primary /1 gb backup, 10 gb)• vBNS network (1 gb)• Defense Research & Exp Net (DREN) (1 gb)• NASA’s NISN and NREN (1 gb ea)• USGS DOI net (1gb)• Qwest DIA ISP service offering (1 gb, 2nd 1 gb)• Cogent ISP service (1 gb)• National Lambda Rail, potentially NLR L3 service.• GEANT2, “European Abilene”, (10 gb, May)• Energy Sciences net (ESnet) (1 gb, becoming 10 gb)• Atlantic Wave distributed peer fabric, (summer 06)

Atlantic Wave over NLR lambda

Sao Paulo (SPB)

LONNYC

WDC

ATL

MIA

AMS

STK

CER

SEA

CHI

AUS

LAXTOK

A-Wave

East Coast confederation of Network Exchange points-Production level IP peering-Experimental “Light Path” services

Atlantic Wave switch partnership with Ciena• Awave switch was in FY05, carried over to FY06 budget. Best

guess, real costs were unknown.• SURA funding of Awave 10G lambdas over NLR is protracted

process due to contractual issues with 4 universities.• Meanwhile, GEANT 10G sonet circuit arriving in May needing to

be terminated using GFP framing. We need to move forward.• We held sonet switch vendor presentations by Ciena, Nortel,

Sycamore, and Alcatel (used by GEANT), got quotes.• Ciena out of blue decided to >>donate<< CoreDirector switch &

sonet interfaces to MAX, do co-development gmpls MOU with DRAGON. Were going donate 5 to Internet2, now MAX instead.

• Talked Ciena into 2nd free switch, more interfaces: 1 production, 1 research. Will need to pay FY07 only for 6 newly released 10G ethernet ints, maint, training, etc.

• Very signification donation to MAX, value $4-500k for 12 OC192 ints, 16 OC48 ints, 20 gige ints in 1 full height & 1 half chassis.

MAX expanded customer colo and disaster recovery sites

• Disaster recovery sites have been a continuing request from members at MAX meetings. People want affordable colo space to host their own gear offsite, yet accessible.

• In response, we have investigated 4 options: trade deals with Northrup Grumman and InterAmerican Development Bank, both new members with colo facilities; visited GWU’s VA facility, and striking a much better price per sq ft and per amp for a lot of space on the second floor of Level3 pop in Mclean. All still in progress, Level3’s would be most convenient if they can bring price down.

• Meanwhile, Max unexpectedly acquired a fully equipped 1200 sq ft machine room across the hall with generator, dual large UPS, overhead power distribution, underfloor dual AC units. Will hold 38 cabinets. Need because MAX test lab out of space, but would like to make available to members also. Quote out to ring out 8400 fiber.

• Would charge folks upfront for standardized cabinets, then lease lambdas back to their institution so have private dedicated bandwidth and looks like part of their network with no MAX L3 in between.

1200 sq ft, can hold 38 cabinets, independent access to outside ramp

One of two large Liebert AC units feeding underfloor cooling

Dual 50 kva UPS with generator feeding overhead dual 208 VAC power rails

That’s fine for folks who can use DR in the DC area, but what about farther out?

• Have begun talks with UMBC about a separate machine room space to be leased to MAX that could hold around 28 cabinets.

• Location (40 miles from College Park) is good, already on new Baltimore dwdm ring so can provision colo lambdas.

• UMBC was original UMD partner in “BWI gigapop” in vBNS days (97), so long history of good working relationship.

• Power is good, space is good, but they need to install additional cooling for new space. Study underway, hopefully reasonable cost

• Intend to have standardized offering both machine rooms for about $1500/cabinet one-time plus $500/cabinet/ month. Wanted to be free as benefit of membership, but needed recoup lease, maint, etc

• Will likely lease private lambdas, mpls tunnels over shared 10G.• If can have online summer (8400) and early fall (UMBC), hoping to

have good uptake in cabinet leases. Are you interested?

MAX Video Lab/Access Grid node• DRAGON has successful ongoing work with ISI/E and others

for development of UltraGrid video codecs for HD over IP.• This work now is crammed into MAX “bank vault” conf room.• Also, we have lot of requests from MAX membership (eg, Holo-

caust Museum this week) for help with video projects like multicasting events. Right now, nowhere to test against.

• Idea is to take old MAX lab after equipment moved to new machine room, convert it to video lab and access grid node. Already well equipped with separate ac, good power, carpet and good lighting. Hope to have new video lab in by fall.

Ladan Gharai of ISI and Chris Tracy of MAX compare parking permitsover high def (“better than life”)

Recruiting for Engineering and Operations Advisory Group

• Ongoing MAX Strategy and Research Advisory Groups.

• Have had Core Engineering Group for years, but that needs to be expanded and meet more regularly (quarterly).

• Want to offer chance for techies to help us plan future MAX engineering projects and directions.

• Caution: this is not for management folks and those wearing suits. It will be highly technical. We hope that the lead network engineers of each institution will be interested.

• The intent is for some sessions to have presentations of plans, others to have brainstorming sessions on selected topics, and periodically to have invited vendor presentations.

• Interested? Please mailto://magorian@maxgigapop.net.

Thanks!