CEN Network Technology Briefing – July 2006. Briefing Agenda Describe UConn’s Leadership in...

CEN Network Technology Briefing – July 2006

Briefing Agenda

• Describe UConn’s Leadership in State, National and Regional Advanced Research and Education Networks– Connecticut's Optical Network Backbone and Architecture– Discussion of UConn's role in providing service to CEN users

• Overview of network content initiatives in K12, higher education and government on these networks (online learning, video, e-portfolio, etc)

• The relationship between the CEN, Internet2, the NOX, Abilene, NEREN and the National Lambda Rail

CEN Services for K12 & Libraries

• Every School district gets an optical drop• On Network Peering to all other CEN sites• Primary Internet Service Provider• Internet2• Firewall• Child Protection Filtering• Domain Name Service• Generally redundant links to each site

CEN Services for Higher Eds

• Redundant Optical drop to every campus

• On Network Peering to all other CEN sites

• Optional Commodity Internet Services

• Optional Internet2 services

• Optional access to NEREN fabric

• Future video, disaster recovery services

CEN Paying Customer Connectivity

• Who’s on Now:– UConn (8)*– CSU (5)*– CommTech

System *– Charter Oak State*– Albertus Magnus*– Yale *– Trinity *– Wesleyan *– UNH *– Conn College *– USCGA *– Rensellaer *

– Sacred Heart *– U Hartford *– Fairfield *– Quinnipiac *– Mystic Aquarium *– Vbrick *– American School for

the Deaf *– Connecticut Public

Television *– St. Joseph’s *– Mitchell *

• Who is next:

– St. Vincent’s – Commtech (4)– U Bridgeport– Lyme Academy– Williams School

CEN Technologies

• Optical backbone on leased dark fiber

• CWDM on congested fiber paths

• Ethernet based Network– Large frame size capacity (MTU of 9216)– MPLS Enabled Core for Layer-2 cut-through– IP Multicast– Capacity to deploy IP v6 overlay

CEN Dark Fiber Backbone

• Fibertech Networks - – “On Network” Dark – Existing backbone areas where

CEN purchased by the pair– “Lateral Build” Dark – 12 strands built for CEN with no

electronics– Erate Leased Ethernet – Built for CEN as a GBIC

based ethernet service

• Singlemode Fiber, SMF28– LX/LR (<10 km) ZX/ER (10>70 km)

BGP RR

BGP RREast Hartford

DOIT

BGP RR

MansfieldUConn

BGP RR

LitchfieldTroop A

STAMFORDUConn Stamford

WEST HAVENQwest NEW LONDON

Qwest

ENFIELD

HartfordUConn Law

New BritainCCSU

WaterburyRowland Center

Ansonia

BridgeportTroop G

SouthburyTroop B

DanburyWCSU

Meriden Norwich

Middletown

HamdenSCSU

BGP RREast Hartford

DOIT


Current 1 Gbps BackboneCurrent Backbone to be upgraded to 10 GbpsBackbone Under Construction

Hub Site Types:

• Telecom POPS (2)– West Haven, New London

• State Police Locations (4)– Meriden, Southbury, Litchfield, Bridgeport

• College Data Centers (9)– Danbury, Hamden, Hartford, Storrs, Norwich,

Middletown, Stamford, Enfield, New Britain

• Borrowed Space (3) – Ansonia, Waterbury

Hub Site Specs

• Design with short fiber lateral before fiber diversity, preferably only building entrance

• Type A Sites (Critical & typically w/ 10G)– Powering

• 4 hours battery with automatic generator backup• 8 hours battery

– Assured 7x24 Access• Type B Sites (Backup Service only)

– 8 hours battery– Less favorable access conditions

BGP RR

BGP RREast Hartford

DOIT

BGP RR

MansfieldUConn

BGP RR

LitchfieldTroop A



Qwest

ENFIELD

HartfordUConn Law

New BritainCCSU


Ansonia

BridgeportTroop G

SouthburyTroop B

DanburyWCSU

Meriden Norwich

Middletown

HamdenSCSU

BGP RREast Hartford

DOIT



A

A A A

A

A

A

A

A

A

A

B

B*->A

B*->A

B

A*

A*

A*

A

A

New London

Ansonia

West Haven

Waterbury

Meriden

Backbone Architecture• Massive over-provisioning to allow multiple link failures with no service impact,

typically 10G on primary backbone• Physical and logical meshing implemented where possible• 9216 MTU Size on all core links• MPLS Tag Switching on all interfaces

– MPLS TTL Propogation disabled except for troubleshooting• All MPLS enabled devices in OSPF Area 0 on all interfaces• BGP Peering for VPNV4 routes only to 5 geographically separated route reflectors• No policy routing, ospf weighting or access lists if possible (let traffic flow its default

path)• Prefix Management

– Global routing table only for on-network connectivity– All customer routes in virtual routing tables– Global multicast only to support MPLS MDT trees– Customer networks also prefer to use OSPF in VRF’s, not using area 0

• Failure Responsiveness– Link State notification on all backbone links should force immediate routing convergence– Longest failures should be based on BGP timers

Fiber Tributary Design

• Higher Education Sites

Hub Site

Hub Site

Higher Ed

Higher Ed

Higher Ed

Higher Ed

GigE LX or ZX

Typically 10GigE

Hub Site

Hub Site

K12 Site

GigE LX or ZX

Typically 10GigE

Fiber Tributary Design

K12 Site K12 Site

K12 Site

• K12 Site Design

Tributary Design

HIGHER ED SITES

• 7000 series software based routers

• OSPF routed /30’s per port• Each campus dual-homed to

two hub sites• MPLS runs to the edge device• >1500 MTU• BGP to the edge

K12 SITES

• 3550 series L3 switches• OSPF Shared /28’s on

backbone vlan• Up to 4 (6) sites per tributary

between two hub sites• No MPLS• 1500 MTU• No BGP

Backbone Construction

• t

Level(3) Conduit Route

• 130 Mile state controlled duct

• 108 Strand Cable Installed

• 96 Singlemode

• 12 LEAF

• 48 Spliced through

• We own the cable

Firewall, Filtering & Server Block

K12 -FWG43 B – VRFAggregate for “inside

routes ”

ENRT043D

ENRT043D

Checkpoint FirewallFWG-43B

NET-FWG1 – VRFAggregate for “inside

routes”

ISP- ONNET(VRF table for all on-network CE

addresses )

ISP-WILTEL -HTFDVRF For Wiltel ISP

Routes(default route only )

Wiltel GigE

G2/12

K12 -East VRFDark Fiber

Connected K 12 Sites

ISP -QWEST -VRF For Qwest ISP ENRT156 H Default

Route Only

ISP -QWEST - NLVRF For Qwest ISP ENRT095 H Default

Route Only

ImportWeight

3500

ImportWeight

2800

Import

Weight3000

Import Weight

1000

ImportWeight

750

K 12-FWG 43A – VRFAggregate for “inside

routes”

ENRT043E

ENRT043E

Checkpoint FirewallFWG - 43A

NET-FWG1 – VRFAggregate for “inside

routes”

ISP -ONNET(VRF table for all on -network CE

addresses )

ISP-WILTEL -HTFD

VRF For Wiltel ISP Routes

(default route only )

K 12- West VRFDark Fiber

Connected K 12 Sites

ISP -QWEST -VRF For Qwest ISP ENRT156 H Default

Route Only

ISP- QWEST -NLVRF For Qwest ISP ENRT095 H Default

Route Only

Import

Weight3500

ImportWeight

2800

ImportWeight

3000Import Weight

1000

Import

Weight750

SBC -ATM for SBC Opteman

Links

ENRT-FWG43 BCustomer K 12 Router

6509 Sup II / Does not run MPLS

Runs BGP /OSPF

ENRT-FWG 43ACustomer K 12 Router

6509 Sup II / Does not run MPLS

Runs BGP /OSPF

Gig Link between “inside” routers shares routes across for redundancy

CEN Firewall Group VRF Import /Export redundancy

ISP-INTERNET 2-NOX

VRF For Internet 2 Routes

(12,000 routes)

ISP -INTERNET 2-NOX

VRF For Internet 2 Routes

(12,000 routes)

K12 -SBCVRF for

Opteman Links

BGP Peers from “Outside” VRF NET -FWG1

where the Internet routes are mixed in to these two “inside routers” . Default route is sent in to

the inside via BGP and customer network prefixes are sent out . Import maps based on

BGP communities tagged on the inside VRF’s

assure the correct return path through the correct firewall from the ISP vrf’s for stateful

inspection .

FWG 43A

DNS #1URL ServerWhatsUp

FWG 43B

DNS #2 URL Server #2

Filtering, Firewall, Server Block

• Design for Full redundancy– Working towards no customer downtime when

a cluster fails or goes off line– Building a business continuity function so

East Hartford can go away without customer impact

Servers:

• Cenmon (Cricket, techsupport site, log server, DNS)

• N2H2 Admin & N2H2 URL Servers (2)• TFTP/FTP• DNS Servers (2)• Radius Servers (2)• VOIP Server• Firewall Management Station

Internet Services Architecture

• Currently 4 Commodity ISP’s– Wiltel Hartford – 1 Gbps – Newark, NJ– Qwest New London – 622 Mbps – Boston, MA– Qwest West Haven – 622 Mbps – New York, NY– NEREN/OSHEAN – 1 Gbps – Boston, MA

• 2 Paths to Internet2/NOX– NEREN Storrs to NOX – 1 Gbps– Qwest New London – OC3

BGP RR

BGP RREast Hartford

DOIT

BGP RR

MansfieldUConn

BGP RR

LitchfieldTroop A



Qwest

ENFIELD

HartfordUConn Law

New BritainCCSU


Ansonia

BridgeportTroop G

SouthburyTroop B

DanburyWCSU

Meriden Norwich

Middletown

HamdenSCSU

BGP RREast Hartford

DOIT



A

A A A

A

A

A

A

A

A

A

B

B*->A

B*->A

B

A*

A*

A*

A

A

INTERNET PROVIDER DRAINS

ISP Architecture

• All ISP routing entities (VRF’s) can run to nearest ISP egress point in event of cohesive network collapse.

• Try not to rate limit in any instance, customers allowed to burst within reason

• Goal is zero customer-impacting downtime

Internet Provider Load Balance

• Qwest WH• Connecticut State

University

• Community Colleges

• UConn Health Center

• CIR = 135 Mbps @ $39/mbps/mo

• Backup for Wiltel

• Averaging 135-140 mbps peak

• Qwest NL• All other UConn


• Backup for West Haven

• Averaging 180 Mbps peak

• Wiltel Htfd• All K12 & Libraries

• All other higher ed campuses


• Backup for Qwest links

• Averaging 600 Mbps peak

These are our provider costs, not including salaries, benefits, program management, NEREN, collocation, etc. Please consider confidential!

A Revolutionary

Idea in Networking

“Old North Church Project”

Vendor Fiber Routes

Northeast Research and Education Network Proposal

NEREN NYC32 Avenue of the Americas

NYSERnet Colocation SpaceNew York, New York

NEREN CAMBRIDGENorthern Crossroads Colocation Space

300 Bent St

NEREN HARTFORDState of Connecticut Data Center101 E. River Dr, East Hartford

NEREN STORRSUniversity of ConnecticutRoute 44, Mansfield, CT

NERENPROVIDENCE

275 Promenade

NEREN SPRINGFIELD1 Federal St

NEREN WORCESTER474 Main St

NEREN ALBANY 194 Washington St

NEREN ROCHESTER-

NEREN SYRACUSE-

NEREN BUFFALO-

Connecticut, Rhode Island and Massachusetts have purchased the route from Manhattan to Cambridge through Stamford, Storrs, Providence, Springfield and Albany for the Old

North Church Project

NEREN Geography

32 Avenue of the Americas, NYC

601 West 26th Street, NYC60 Hudson Street, NYC230 Congress Street,

Boston300 Bent Street, CambridgeAlong Mass Pike, LeeAlbany375 Promenade,

Providence450 Main Street, Worcester

54 Meadow Street, New Haven

RT 44, Grand Union, Storrs101 East River Drive, E.

HartfordStamfordPomfret

NEREN Technology

• Currently Gigabit Ethernet from Hartford to Boston to Springfield

• DWDM Multiplexing Planned– 32 lambdas of minimally 2.5 Gbps capacity– Likely 10Gbps Ethernet lambda deployment– Some interest in Infinera O-E-O products

• Sparse network utilizing state infrastructure for local distribution

CEN OPERATIONAL THOUGHTS

• When in doubt, broadcast it out– Internal staff email list [email protected]– Customer list: [email protected]

• No core changes without discussion

• Our change window is 5-7 AM with 5 day customer notice

• Edge sites more tolerant of customer requested timing– Remember K12 Daisy-chain convergence issues.

Questions/Contact Information

John Vittner

860-622-2241

[email protected]

Robin Brown

860-622-2139

[email protected]

CEN Network Technology Briefing – July 2006. Briefing Agenda Describe UConn’s Leadership in...

Documents

Transcript of CEN Network Technology Briefing – July 2006. Briefing Agenda Describe UConn’s Leadership in...