1 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Design and...

1NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007

Design and Implementation of

the REANNZ KAREN networkJörg Micheel


Outline for this talk

• Network design goals for KAREN• Layer 1+2+3 network architecture• Network services and implementation• International transit network design• Network performance• Checklist for KAREN connectors (REANNZ members)• Summary and references


MoRST/REANNZ/KAREN design goals

• A high performance network for the NZ R&E community!• 10 Gbps capable backbone interconnecting all major cities in NZ• Access speeds at 1 Gbps and 10 Gbps• Unconstrained end-to-end performance at (multi-)Gigabits/second• Tailored on-demand performance for specific applications or

experiments (bandwidth, delay, jitter)• International connectivity at 155 Mbps to AU, 622 Mbps to US• New services: multicast, IPv6, Jumbo frames (9000 Bytes MTU)• Virtual Private Network functionality for members• Telco-grade implementation and network management• Security, redundancy, high availability• Range of network measurement facilities (wire tap, NetFlow,

SNMP data collection, active measurement) and development environment

• Most importantly: stick to budget and timelines!!!


L1/L2/L3 Network Design

CityLink Auckland VectorNorth Shore

10Gb

10Gb

Forest ResearchRotorua

Innovation Waikato

10Gb

10Gb

TCL POP Napier 10Gb

Hawkes Bay ResearchCentre

Havelock North10Gb

10Gb

Inspire.NetPalmerston North

10Gb

Massey UniversityPalmerston North

10GbCityLink

Wellington 10Gb GNS Lower Hutt

10Gb

10Gb

TCL POPNelson

10Gb

10GbCanterbury University

10Gb

10Gb AgResearchInvermay

= Nortel DWDM Connection

= Fibre Connection

10Gb

10Gb

Hort ResearchMount Albert

CityLinkWellington

= Fibre Patch Lead

REANNZ Logical Network DiagramVersion 1_6

South Island Build15th March 2006

Lincoln University

Otago University

10Gb

VectorNorth Shore

Queenstown

Invercargill

Future10Gb

10GbTo be Replaced

Future10Gb

Future10Gb

• L1 Network core as rings on TCL OPTera DWDM

• L1 Dark fiber spur to neutral POP and AAP

• L2 Nationwide network based on Extreme X450a and BD10K

• L3 using Juniper M320 in Auckland and Wellington

• Note: 10GigE WANPHY is 9.287 Gbps!

• ANOPS management network based on TCL PIP service and CISCO 2801


Extreme Networks Black Diamond – metro core switch

• Black Diamond 10808 (BD10K)• 22 rack mount units• 1280 Gbps capacity (blocking)• Up to 48 10-Gigabit ports• Up to 480 10/100/1000 ports• Powerful VLAN, Virtual router Layer2

and Layer3 capabilities• Proprietary EAPS link-protection

protocol provides continuity in case of fiber cut

• L2/L3 Quality-of-Service• L2/L3 hardware filtering and priority• Jumbo frames at 9212• High availability, hardware

redundancy


Extreme Networks Summit X450a – edge switch

• X450a-24t with 24 ports 10/100/1000 copper, four combined SFP GigE ports

• X450a-24x with 24 ports 1-GigE SFP, four combined 10/100/1000 ports

• Optional dual 10-Gigabit Ethernet uplinks• 1 RU form factor• 160 Gigabits-per-second capacity• 65 million packets-per-second forwarding performance• Stacking capability with XOS 11.7 (April 2007)• Other features similar to Black Diamond series


Juniper Networks M320 Multi-service Edge Router

• 320 Gbps switching capacity• 8 FPC slots with 20 Gbps FD

capacity• ½ rack size• 32 PICs per chassis• 10 GigE capable

PULL PULL

MENU

ACO/LT

ENTER

0

123

POWERFAIL OK

0

123

SIBSFAIL OK

OKFAIL MASTER

RE 1RE 0

M320

FPC 0FAIL OK

FPC 1FAIL OK

FPC 2FAIL OK

FPC 3FAIL OK

FPC 4FAIL OK

FPC 5FAIL OK

FPC 6FAIL OK

FPC 7FAIL OK

FPC 3

YE

LLO

W

ALARMS

RE

D

AU

X

CO

NS

OLE

AU

X

CO

NS

OLE

ET

HE

RN

ET

YEL=10MbGRN=100Mb

ACTIVE

HOST 0

ET

HE

RN

ET

YEL=10MbGRN=100Mb

ACTIVE

HOST 1

10GBASE-XENPAK

TX

RX

10

GB

AS

E-E

R

STATUS

ONLINEOFFLINE

FPC 1


L2/L3 design

TelstraClearAS38022

Auckland M10i

Seattle M10i

Sydney M10i

VerizonBusinessAS38018

iBGP

622

MB

its/s

ec

155 MBits/sec

iBG

P

Auckland M320anr1.reannz.net

Wellington M320anr2.reannz.net

eBG

P

eBG

P eBG

P

Advanced Network10 Gigabit Backbone

IS-IS, iBGP

IS-IS

, eBGP

Gigabit Ethernetto Citylink

Auckland PoP

Gigabit Ethernetat Canterbury PoP

University ofAucklandAS9431

Gigabit Ethernetat Rotorua PoP

Scion ResearchRotorua AS TBN

University ofCanterbury

AS9432

VLAN EAST

VLAN WEST

IS-IS

, eBG

PVLAN REAST

VLAN RWEST

• Connector joins KAREN via dark fiber

• Switch access into one or more VLANs

• BGP peering with core

• L2 packet switched data nationally

• L3 routing overseas


KAREN Service Matrix

IPv4 IPv6 JumboDomestic Unicast L2 switch + BGP RR Same dual VLANDomestic Multicast PIM-SM, MSDP Future n/aInternational U+M L3 routing Verizon Same Default

University of AucklandAS9431 130.216.0.0/16

eBG

PAN-TCL

AS38022 210.7.32.0/21

AN-VZBAS38018 210.7.47.0/24

210.7.32.10

210.7.32.1

210.7.36.1

210.7.36.3


Internet Exchange model (L2 switching + BGP Route Reflector)


AN-TCLAS38022 210.7.32.0/21

210.7.32.10

210.7.32.1

University of CanterburyAS9432 132.181.0.0/16

210.7.32.12

eBGP

eBG

P

“Switch – don’t route”“Peer with two – route with many (others)” scalable Internet Exchange model


Challenges of a L2 networkCityLink

AucklandVector

North Shore

Forest ResearchRotorua

Innovation Waikato

TCL POP Napier

Inspire.NetPalmerston North

Massey UniversityPalmerston North

CityLinkWellington

GNS Lower Hutt

TCL POPNelson

Canterbury University

Hort ResearchMount Albert

CityLinkWellington

Lincoln University

VectorNorth Shore

Queenstown

Invercargill

AgResearchInvermay

Otago University

REANNZ Advanced NetworkGeneric VLANs, illustratingTopology and Geography

Hawkes BayResearch CentreHavelock North

= VLAN EAST

= VLAN WEST

• Redundant links will be pruned (Spanning Tree, etc), creating a star topology

• Only difference between L2 resilience protocols is speed

• Issue: capacity not utilised

• Issue: shortest path• Issue: protect all VLANs

• Solution: VLANs EAST/WEST for public IP services

• Solution: Extreme EAPS for protection


KAREN Multicast – two options for connectors

• Bootstrap as part of KAREN multicast cloud (quick start for small sites, no MSDP, but doesn’t scale)

• Create your own multicast domain (requires MSDP, scales well)


AN Rendezvous PointANYCAST 210.7.36.192/32

PIM-SMborder

PIM-SM

IGMP

IGMP

IGMP

RP

Large sites on the Advanced Networkdeploying their own RP will implement a PIMborder at their AN router and run an MSDPpeering session with each of the RPs of the

AN. All routers between the end user and theAN remain PIM-SM and IGMP enabled.

RP

RP

MS

DP

MSDP

LOOPBACK 210.7.36.193/32 LOOPBACK 210.7.36.194/32

PIM-SM


Multi-Protocol BGP and routing tables

Table Used for Filled in byinet.0 IPv4 unicast routing BGP, ISIS, … AFI=1 SAFI=1inet.1 IPv4 multicast forwarding cache MSDP, …inet.2 IPv4 multicast routing (RPF) M-BGP AFI=1 SAFI=2inet.3 IPv4 MPLSinet6.0 IPv6 unicast routing M-BGP, ISIS, …AFI=2 SAFI=1inet6.1 IPv6 multicast forwardinginet6.2 IPv6 multicast routing (RPF) M-BGP AFI=2 SAFI=2

M-BGP identifier


KAREN International

Aucklandanr1.reannz.net

Wellingtonanr2.reannz.net

AN-TCL AS38022TelstraClear

iBGP

CENIC AS2153CalREN-HPR

PNWGPAS101

Internet2 ITNAS11537:2501

AARnet, AustraliaAS7575:AS-CUSTOMER

AS38018 at Auckland, New Zealand

import AS38022 domestic customer routes as AS38018:38022

export AS38018:7575 AARnet routesexport AS38018:101 PNWGP routesexport AS38018:2153 CENIC routesexport AS38018:11537 Internet2 ITN routesexport AS38018 prefix 210.7.47.0/24

AS38018 at Sydney, Australia

import AS7575:AS-CUSTOMER routes asAS38018:7575

export AS38018:38022 Advanced Network NZdomestic routesexport AS38018 prefix 210.7.47.0/24

AN-VZB AS38018Verizon Business

iBGP

eBG

P

eBG

P

eBG

P

eBG

P

eBG

P

AS38018 at Seattle, WA, USA

import PNWGP routes as AS38018:101import CENIC routes as AS38018:2153import Internet2 ITN routes as AS38018:11537

export AS38018:38022 Advanced Network NZdomestic routesexport AS38018 prefix 210.7.47.0/24

AU

routes US rout

es

AS38022 at Auckland, New Zealand

import ALL routes from AS38018

export domestic customer routesexport AS38022 prefix 210.7.32.0/21

• Separate to domestic KAREN, but co-joined

• As a static 3-point transit network has to implement all services (IPv4/IPv6 uni- and multicast, jumbo frames)

• Routing policy ensures traffic flows between NZ and overseas peers (but not between other peers)

• Pacific Wave landing point in Seattle poised for peering expansion

• Bulk of 9K routes from Internet2 ITN


REANNZ POP


KAREN performance tests

• Network commissioning in October and November successfully demonstrated capacity, delay and jitter parameters

• Bandwidth tests carried out as 1 Gbps VLANs POP-to-POP

• All L2 components stressed at or near capacity limits (see next slide for example)

• Delay and jitter tests carried out as RTT measurements using hardware loopbacks


KAREN performance tests (as shown on WAND weathermap)

See recorded animations at http://erg.cs.waikato.ac.nz/weathermap/ for other tests carried out during November and December.


KAREN delay and jitter tests

CityLink AucklandScion Research

Rotorua

TCL POP Napier

GNS Lower Hutt

CityLinkWellington

1 2 3 4 A B 5 6 7 8

G20X61011

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

20

1

DS

10

11

10G6X61 050

DS

1

2

3

4

5

6

1

2

3

4

5

6

62001

CONSOLE

MGMT

COMPACTFLASH

MSM-1XL60016

MS

TR

EN

V

SY

S

ER

R

62 001 62001 62 001 62 00162 001 62 001G20X

61 011

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

20

1

DS

10

11

10G6X61 050

DS

1

2

3

4

5

6

1

2

3

4

5

6

CONSOLE

MGMT

COMPACTFLASH

MSM-1XL60016

MS

TR

EN

V

SY

S

ER

R

CONSOLESummit X450a-24t

TM

1 3 52 4 6 129 11107 8 13 1514 16 17 19 2118 20 22

Shared Ports

23 24 21x 22x 23x 24x

STACK NO.

Solid ON = Link

FAN

PSU-I

PSU-E

MGMT10G

Stack

1

2

1

2

Blinking = Activity


TM

1 3 52 4 6 129 11107 8 13 1514 16 17 19 2118 20 22

Shared Ports

23 24 21x 22x 23x 24x

STACK NO.

Solid ON = Link

FAN

PSU-I

PSU-E

MGMT10G

Stack

1

2

1

2

Blinking = Activity


TM

1 3 52 4 6 129 11107 8 13 1514 16 17 19 2118 20 22

Shared Ports

23 24 21x 22x 23x 24x

STACK NO.

Solid ON = Link

FAN

PSU-I

PSU-E

MGMT10G

Stack

1

2

1

2

Blinking = Activity

1 2 3 4 A B 5 6 7 8

G20X61011

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

20

1

DS

10

11

10G6X61 050

DS

1

2

3

4

5

6

1

2

3

4

5

6

62001

CONSOLE

MGMT

COMPACTFLASH

MSM-1XL60016

MS

TR

EN

V

SY

S

ER

R

62 001 62001 62 001 62 00162 001 62 001G20X

61 011

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

20

1

DS

10

11

10G6X61 050

DS

1

2

3

4

5

6

1

2

3

4

5

6

CONSOLE

MGMT

COMPACTFLASH

MSM-1XL60016

MS

TR

EN

V

SY

S

ER

R

EXAMPLEAssessing the switch-switch

delay between Napier andGNS Lower Hutt

REANNZ PoP TSNPVector North Shore

SmartBits LAN-3321A4x 1-Gigabit-Ethernet

Measurement process:

1) Set up loopback cables at Napier and Lower Hutt2) Create individual VLANs from ACLD lab toNapier and Lower Hutt cable loops3) Measure packet delay and jitter to each site4) Subtract difference and assess errors


TM

1 3 52 4 6 129 11107 8 13 1514 16 17 19 2118 20 22

Shared Ports

23 24 21x 22x 23x 24x

STACK NO.

Solid ON = Link

FAN

PSU-I

PSU-E

MGMT10G

Stack

1

2

1

2

Blinking = Activity

REANNZ KARENDelay and Jitter Tests

Version 1_215th November 2006


KAREN delay and jitter results

From Route Site Theor Diff Percentanx12-tsnp DIRECT anx1-waun 132.7 2.1 102%

ONEHOP anx11-masc 253.2 5.3 102%

EAST anx13-frir 3599.0 102.3 103%EAST anx22-napr 6291.9 86.4 101%EAST anx23-napr 6307.9 77.0 101%EAST anx19-crof 6564.0 60.1 101%EAST anx17-nfuw 10357.1 151.8 101%EAST anx2-lmtn 10673.9 267.2 103%EAST anx3-cscc 15894.7 420.4 103%EAST anx20-linu 16324.0 385.4 102%EAST anx4-otun 21093.0 -248.4 99%EAST anx21-resz 21113.0 -55.2 100%

WEST anx14-innp 2008.8 133.8 107%WEST anx15-hepn 7537.6 450.4 106%WEST anx16-mass 7657.4 438.3 106%WEST anx2-lmtn 9578.2 535.4 106%WEST anx18-neln 12308.9 544.0 104%WEST anx3-cscc 17274.6 742.5 104%WEST anx4-otun 22043.7 746.1 103%

RTT switched (in usecs)


KAREN Connectors 101 (and shopping list)

• KAREN is a Tier 1 network – you need to behave like a Tier 2 – control your own routing (policy).

• Consider the services you want: IPv4 unicast, multicast, IPv6, Jumbo• Are you a heavy hitter ? Thinking of 10 Gbps ?

Router that speaks BGP, holds 20K+ routes and does 1 Gbps• If you are a heavy hitter, you need VLAN support and 40K+ routes• If you want jumbo frames, you need VLAN support and 40K+ routes• If you want multicast, you need PIM-SM, preferably MSDP and M-

BGP• If you want IPv6, you need M-BGP and space for even more routes

IPv4 IPv6 JumboDomestic Unicast L2 switch + BGP RR Same dual VLANDomestic Multicast PIM-SM, MSDP Future n/aInternational U+M L3 routing Verizon Same Default


KAREN Connectors to date

• About a dozen connectors at around 15 sites, wide range of equipment

• CISCO 6500 series edge routers• Allied Telesyn AT-9924Ts• Juniper M and J series – J6350• Linux PC and Quagga Routing Suite• Check Point Firewall on Nokia platform


2007 outlook

• Connectors, connectors, connectors!• Access solutions for schools and other non-BGP speakers• International IPv6 peering• Peering with US FedNets; China, Japan R&D networks• IPv6 DNS• Better solution to the 2/4/8 peering problem for complex

sites• Pushing traffic around EAST/WEST for dual attached sites• L2 PIM-SM snooping (on top of IGMP snooping)• More security, core hardening• Stacking support in Napier and 10 Gbps services to

Havelock North• Measurement infrastructure (active and passive) –

capability build fund


Summary

• KAREN creates a fast lane for the R&E community, inside New Zealand and with overseas R&D networks.

• It offers a range of new services previously unavailable or inaccessible in New Zealand, such as multicast, IPv6.

• It offers a test bed for novel tools and applications.• Performance is stunning – go and use it!


Acknowledgements and references

• REANNZ KAREN http://www.karen.net.nz/• WAND weather map

http://erg.cs.waikato.ac.nz/weathermap/• A cast of dozens of hands at TelstraClear and JazzTech

• Questions: please contact myself or David Brownlie and Clayton Ejiofor at REANNZ.

Thank you!

1 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Design and...

Documents

Transcript of 1 NZNOG 2007 – Inspire.Net / Massey University, Palmerston North, February 1st, 2007 Design and...