CANARIE http://www.canarie.ca

CA*net 4 Design DocumentLast Revised April 22 2001

Version 1.20OBGP documentation and latest version of this

document can be found at

http://www.canet3.net

Bill.St.Arnaud@canarie.caTel: +1.613.785.0426

The Concept for CA*net 4 Conventional optical networks are built on the paradigm that a central entity

has control and management of the wavelengths It therefore must have control of the edge device for the setup and tear down of

the wavelengths Will central control and management scale to millions of edge device and

thousands of optical wavelengths? Customer empowered optical networks are built on the paradigm that

customer owns and controls the wavelengths (Virtual Dark Fiber) Customer controls the setup, tear down and routing of the wavelength between

itself and other customers Customer may trade and swap wavelengths with other like minded customers

ultimately leading to wavelengths as market commodity How do you design a network architecture if the routing and control of

wavelengths is under the control of the customer at the edge? Network is now an asset, rather than a service

Analogy to time sharing computing in the early 1970s versus customer owned computers or client-server computing

Condo Fiber & Wavelengths Condo fiber means that separate organizations own individual strands of

fiber in a fiber cable Each strand owner responsible for lighting up the strand Collectively responsible for sharing costs of maintenance on fiber cable,

relocation etc Condo wavelengths

Number of parties share in the cost of a single strand and that light up with an agreed upon number of wavelengths

Wavelengths are portioned based on percentage ownership With condo fiber and condo wavelengths institutions can treat network as an

asset just like purchasing a computer, rather than a service as today

Research Network Issues Research and Education networks must be at forefront of new network

architecture and technologies Should be undertaking network technology development that is well ahead

of any commercial interest But any network architecture can only be validated by connecting real users

with real applications and must solve real world problems Test networks per se are not sufficient

There is a growing trend for many schools, universities and businesses to control and manage their own dark fiber

Can we extend this concept so that they can also own and manage their own wavelengths?

Will “empowering” customers to control and manage their own networks result in new applications and services similar to how the PC empowered users to develop new computing applications?

CA*net 4 Research Objective To deploy a network architecture where the GigaPOPs and institutions at the

edge manage and control their own fiber and their own wavelengths Condominium fiber and condominium wavelengths

To deploy a novel new optical network of distributed optical IXs that gives GigaPOPs and communities at the edge of the network (and ultimately their participating institutions) the ability to setup and manage their own wavelengths across the network and thus allow direct peering between GigaPOPs on dedicated wavelengths and optical cross connects that they control and manage

To allow the establishment of wavelengths by the GigaPOPs and their participating institutions in support of eScience and grid applications to support true peer-to-peer networking

To allow connected regional and community networks to setup peering relationships with CA*net 4 for collaborative research and education and eScience applications

To partner with private sector in building “carrier neutral” distributed optical Internet exchange facilities across Canada and developing new services in fungible wavelengths to enable customer empowered optical networks

Current View of Optical Internets

Big Carrier Optical Cloud using MPS or ASON for management of wavelengths

for provisioning, restoral and protection

Carrier controls and manages edge devices

Optical VLAN

Customer

ISP

AS 1

AS 2

AS 3

AS 1AS 4

AS 5

UNI

NNI

Customer Empowered Metro Network

Carrier Neutral IX &OBGP switch

City A

City B

City C

Carrier Neutral IX& OBGP switch

Condo Dark Fiber

Condo Wavelengths

Condo Wavelengths

OBGP switch

OBGP switch

Future Optical Networks

Customer A

Customer B

Customer C

Customer D

Customr A elects to cross connect with Customer C rather than D

Massive peering at the edge

Condo Fiber

Condo Wavelength

CA*net 4 Research Areas New optical technologies that support customer empower networking

OBGP, CWDM, hybrid optics and HWDM, customer controlled optical switches

BGP scaling issues Object Oriented Networking

Wavelengths and optical switch treated as an object and method to be incorporate into middleware

Or treated as fungible product Distributed Computing Applications and Grids

Wavelength Disk Drives (WDD) eScience Grids for weather forecasting, forestry management, education, health,

etc

Object Oriented Networking Combines concepts of Active Networks and Grids

See DARPA See Globus

Customer owns sets of wavelengths and cross connects on an optical switch Network elements can be treated as a set of objects in software

applications or grids Complete with inheritances and classes, etc Rather than distributed network objects ( e.g. Java or Corba) distributed

object networks In future researchers will purchase networks just like super computers,

telescopes or other big science equipment Networks will be an asset – not a service Will be able to trade swap and sell wavelengths and optical cross connects

on commodity markets

Advantages of OON With massive peerings to the edge, the loss of one peer is not catastrophic

No need for restoral or protection paths or ring architectures Networks look more like “star bursts” rather than “ring of rings”

See C Labovitz ACM Sigcomm Aug 2000 – massive peering helps faster convergence

May solve problem of scaling large networks Today M carriers building meshed networks to N customers with

resultant M*N2 requirement for wavelengths With OON the global requirement for wavelengths grows at X*N

where X= average number of wavelengths per customer

Example OON Earthquake Visualization Grid

Globus Middleware Begin Establish connection to other grid participants

Network Object – wavelength to STAR LIGHT – Chicago Network Object – wavelength to Research center Amsterdam Network Object – wavelength to SDSC Visualization Computer Network Object – wavelength to Seismology Center Calgary Link objects and create grid

Run Visualization Release Network objects

Globus Middleware End Earthquake Visulization End

Napster OON

University in Canada willing to exchange these wavelengths

Montreal-NY blue NY-Amsterdam red Chicago-Montreal green

Wants these wavelengths Chicago-SDSC - purple SDSC- Hawaii - red Chicago- Chile - yellow

University in Chicago willing to swap these wavelengths

Chicago to SDSC – purple SDSC- Hawaii – Red Chicago to Argonne - Blue

Wants wavelength to these locations

Chicago to Toronto - yellow

“National grand challenge" e-research projects are on the horizon: with the next generation network, interconnecting to school and community networks, Canadian researchers could use the thousands of computers in schools and communities distributed across Canada

Students at schools and ultimately members of the public could be full participants in basic reserach

The next generation research network should be designed to encourage and enable projects such as these

The eScience Vision

Wavelength Disk Drives

Vancouver

Computer data continuously circulates around the WDD

Calgary

Regina

Winnipeg

Ottawa

Montreal

Toronto

Halifax

St. John’s

Fredericton

Charlottetown

CA*net 3/4

WDD Node

eScience Grid

Customer A

Customer B

Customer C

Customer DWDD Grid

Customers autonomously create WDD ring for high

performance applications

Wavelength Disk Drives

CA*net 4 will be “nation wide” virtual disk drive for grid applications

Big challenges with grids or distributed computers is performance of sending data over the Internet TCP performance problems Congestion

Rather than networks being used for “communications” they will be a temporary storage device

Ideal for “processor stealing” applications

OBGP Proposed new protocol to support control and management of wavelengths

and optical switch ports Control of optical routing and switches across an optical cloud is by the

customer – not the carrier – true peer to peer optical networking Use establishment of BGP neighbors or peers at network configuration

stage for process to establish light path cross connects Customers control of portions of OXC which becomes part of their AS Optical cross connects look like BGP speaking peers – serves as a

proxy for link connection, loopback address, etc Traditional BGP gives no indication of route congestion or QoS, but with

DWDM wave lengths edge router will have a simple QoS path of guaranteed bandwidth

Wavelengths will become new instrument for settlement and exchange eventually leading to futures market in wavelengths

May allow smaller ISPs and R&E networks to route around large ISPs that dominate the Internet by massive direct peerings with like minded networks

Opportunity for carrier and industry partners

To participate in a novel new Internet architecture that will allow customers to manage and control their own wavelengths anywhere across the network

Very attractive technology for Tier 2 ISP, research networks and ASPs Yahoo and Cable and Wireless have already started down this path It will allow them to create their own network topologies

To provide a valuable new service for customers that will allow them to reduce Internet transit costs by as much as 75%

To develop new value added services in IX brokering and management To develop new fungible trading services in bandwidth trading and brokering To experiment with new long haul optical technologies that will dramatically

reduce cost of long haul transmission

CA*net 4 Possible Architecture

Vancouver

Calgary ReginaWinnipeg

Ottawa

Montreal

Toronto

Halifax

St. John’s

Fredericton

Charlottetown

Chicago

Seattle

New York

Europe

Customer controlledoptical switches

Layer 3 aggregation serviceOptional Service Available to any GigaPOP

Large channel WDM system

Wavelength Scenarios

Vancouver

Calgary

ReginaWinnipeg

Toronto

Halifax

St. John’s

Seattle

Montreal

Workstation to Workstation Wavelength

University to University Wavelength

CWDM

BCnet

RISQ

GigaPOP to GigaPOP WavelengthCampus OBGP switch

Wavelength Setup

AS 1

AS 2

AS 3

AS 4

AS 5

AS 6

Dark Fiber

Wavelength Object owned by primary customer Wavelength Subcontracted by primary customer to a third party

AS 1- AS 6 Peer

AS 2- AS 5 Peer

2

3

4

5

6

17

8

9

1012

13

14

15

Regional Network

Regional NetworkUniversity

University

ISP router

Wavelength Logical Mapping

AS 1

AS 2

AS 3

AS 4

AS 5

AS 6

Primary Route

Backup Route

AS 1- AS 6 Peer

AS 2- AS 5 Peer

2

3

4

5

6

17

8

9

1012

13

14

15

Regional Network

Regional NetworkUniversity

University

ISP router

Resultant Network Topologies

AS 1

AS 5

AS 6

13

14

15

Regional Network

Regional NetworkUniversity

University

AS 2

2

1

7

ISP router

8

9

5

9

1

2

8

6

7

310

5

12

10OBGP

Potential OBGP Peering

BGP Peering on switches at the edgePacket Forwarding in the core

Possible CA*net 4 Node

4 Channel GbE CWDM to local GigaPOP

Carrier A

Carrier B

OC48 DWDM

OC192 DWDM

8 Channel GbE CWDM to next CA*net 4 node

2xGbE

10xGbE

Optional Aggregating Router

Carrier A OBGP Switch

CA*net 4 switch

Physical Wavelength Configs

OC192 DWDMCarrier B

10xGbE 10xGbE

Carrier A

OC48 DWDM

2xGbE 2xGbE

OBGP links

ORAN B ORAN A ORAN C ORAN D

Dark fiber +CWDM

2xGbE

1

23 4

CA*net 4

5

6

7 8

Logical Wavelength Configs

OBGP links

ORAN B

ORAN A

ORAN C ORAN D

Carrier A

Carrier B

GbE over CWDM

CA*net 4

GbE over 10GbE over OC-192 DWDM

GbE over 2GbE over OC-48 DWDM

1 2 3 4

5

Possible Wavelength Assignment Illustrative purposes only Assume 150 wavelength system across Canada 50 wavelengths assigned to provincial networks based on a number of

criteria including ability to extend wavelengths into provincial network and requirements for high bandwidth applications

ORANs encouraged to extend wavelengths to individual institutions Institutions encouraged to deploy optical switches

On all cross sections a minimum of 100 wavelengths dedicated to CA*net 4 and carrier partners

2 wavelengths dedicated to CA*net 4 layer 3 aggregation service (looks like old CA*net 3)

10 wavelengths (and OCX ports) reserved for temporary applications like Grids or eScience

A wavelength and OXC port bartering and exchange mechanism so that ORANs can swap wavelengths will be an important requirement

Example Physical Architecture CANARIE builds heterogeneous network made from many sources e.g (illustrative

purposes only): dark fiber from St. John to Halifax using ULH 16 channel POS dark fiber condo from Halifax to Fredericton using 16 channel 10GbE Condo wavelengths from RISQ from Edmonston to Ottawa sharing 32 channel 10GbE

system dark fiber from Ottawa to Winnipeg with Onet using 16 channel POS at OC-192 Condo wavelengths from Bell Canada from Montreal to Chicago as part of a 140

wavelength system wavelengths from Telus from Chicago to Winnipeg as part of a 140 wavelength system dark fiber from GT Telecom from Winnipeg to Calgary using 16 channel 10GbE wavelengths from Shaw from Calgary to Vancouver as part of a 32 channel 10GbE

systems wavelengths from 360 Networks from Halifax to London as part of a 400 wavelength

system Wavelengths from Teleglobe from Seattle to Honolulu – Sydney – Tokyo - Seoul

OBGP Variations1. OBGP Cut Thru

OBGP router controls the switch ports in order to establishes an optical cut through path in response to an external request from another router or to carry out local optimization in order to move high traffic flows to the OXC

2. OBGP Optical Peering External router controls one or more switch ports so that it can establish direct

light path connections with other devices in support peering etc

3. OBGP Optical Transit or QoS To support end to end setup and tear down of optical wavelengths in support of

QoS applications or peer to peer network applications

4. OBGP Large Scale To prototype the technology and management issues of scaling large Internet

networks where the network cloud is broken into customer empowered BGP regions and treated as independent customers

OBGP Optical Peering Primary intent is to automate BGP peering process and patch panel process Operator initiates process by click and point to potential peer Original St. Arnaud concept

Uses only option field in OPEN messages Requires initial BGP OPEN message for discovery of OBGP neighbors Virtual BGP routers are established for every OXC and new peering relationships are

established with new BGP OPEN message Full routing tables are not required for each virtual router No changes to UPDATE messages No optical transit as all wavelengths are owned by peer Uses ARP proxy for routers on different subnets

Wade Hong Objects concept Uses an external box (or process) to setup optical cross connects SSH is used to query source router of AS path to destination router Each optical cross connect is treated as an object with names given by AS path Recursive queries are made to objects to discover optical path, reserve and setup NEXT_HOP at source router is modified through SSH End result is a direct peer and intermediate ASs disappear Requires all devices to be on same subnet

OBGP Optical Transit Wavelengths are under control of another entity who has temporarily allowed

them to be available for transit Viagenie – Marc Blanchet and Florent Parent

Designed specifically for optical transit applications Uses MBGP and establishes new address family for OBGP Community tags are used to advertise availability of optical paths as part

of NLRI and COMMUNITY TAG Reservation and setup is done by advertising update NLRI message Exploring using CR-LDP & RSVP-TE with AS loose routing for path

reservation and setup Changcheng Huang

The same NLRI message is sent back and forth and modified to indicate first availability of wavelengths, reservation and setup

Over rides loop back detection in RIBS for advertised NLRI messages

Target Market for OBGP University research and community networks who are deploying

condominium fiber networks who want to exchange traffic between members of the community but who want to maintain customer control of the network at the edge and avoid recreating the need for aggregating traffic via traditional mechanisms

E.g. Ottawa fiber build, Peel County, I-wire, SURAnet, G-Wire, CENIC DCP, SURFnet, etc etc

Next generation fiber companies who are building condominium fiber networks for communities and school boards and who want to offer value added fiber services but not traditional telcommunications service

E.g. C2C, Universe2u, PF.net, Williams, QuebecTel, Videotron, etc Next generation collocation facilities to offer no-cost peering and wavelength

routing Metromedia, Equinix, LINX, PF.net, LayerOne, Westin, PAIX, Above.com,

Colo.com, etc etc Over 500 Ixs and carrier hotels worldwide

OBGP Peering

Possible technique for allowing automatic peering at IXs between consenting ISPs

External routers are given control of specific ports on the OXC The router that controls switch can act as an optical route server notifying all

peers of any new consenting OBGP peers External routers signal to each other if they wish to setup direct optical

connection Choice of partner can be based on size of traffic flows Partners can be changed through a routing flap

Only see each other’s customers routes – not the default core

OIX using OBGP

AS 100160.10.10.0

AS 200170.10.10.0

AS 300180.10.10.0

AS 400190.10.10.0

Institution A

Institution B

Institution C

Institution D

Figure 10.0

Switch Ports are part of institution’s AS

Transport Architecture Heterogeneous transport architectures used on backbone links Type of transport architecture on each link determined by length of link between O-

ADMs, GbE-ADMs or OBGP switches, requirement for optical repeaters or regenerators, etc

Examples: 8 or 16 channel GbE used on short haul links (up to 2000 km) between OADMs or

OBGPS; or OC-192 Ethernet over SONET with multiplexed 10 single GbE or trunked 10GbE;

or Proprietary 8 channel 2 x GbE multiplexed into OC48 optics with FEC wrapper

Repeaters: GbE or 10GbE 2R transceivers every 50-80 km combined with GbE or 10Gbe 3R

switches every 200 – 400 km; or Traditional EDFAs at 1550nm every 50- 80 km with OC-192 regenerators at every

200-400km; or All optical broadband: Counter rotating Raman amplifiers, multi band EDFAs,

EFFs, dispersion correction fiber, etc

Tributary Architecture

Customer can connect through OADM,Gbe-ADM, direct to OBGP switch or through CA*net 4 router

Customer access link is either GbE or trunked 10GbE (I.e. 10 separate GbE channels

In future customer will have a choice of protocols, but for now GbE will be basic standard across the network

Switch Architecture Low speed MEMs or similar capacity switch

Could also use non blocking GbE switch Switch can also be distributed across an optical network using GMPLS or

ODSI Each switch component can be controlled by a socket/port by any external

network element with appropriate security mechanisms If OXC used for traffic engineering or QoS then controlling router manipulates

both input and output ports If OXC used for distributed peering then participating AS only owns either

INPUT or OUTPUT ports Eventually switches can also support optical trunking of many optical paths Switch commands are kept very simple, leaving all complexity to OBGP

messages Switch does not know or care the direction of the wavelength – that is

established with OBGP protocol