Connecting to an IXP

IX Etiquette and Hygiene

• Connect using a layer 3 device

• Don’t proxy ARP• No CDP, RIP, EIGRP, OSPF, ISIS– https://ams-ix.net/technical/specifications-

descriptions/config-guide

• Don’t steal default

• Don’t leak the IX prefix to the Internet

• Do use consistent announcements

• Do register prefixes in an Internet Routing Registry

2

Filtering Announcements

• Only send infrastructure and customer routes– Can use community tagging to easily identify them

• Filter what you accept– Route filters (use Routing Registry data)– AS path filters– Maximum prefix count

• Minimum prefix size– Typically a /24 for IPv4 (/48 for IPv6)– May special case host routes for blackhole

3

PeeringDB

• https://www.peeringdb.com

• Identifies your AS number• Provides contacts for NOC– Very useful when using a MLPA and need to contact peer

• Shows which facilities you use for peering– IP numbers in use at those facilities

• Brief description of who you are, how to contact you, your traffic levels, type of customers, your peering policy

4

PeeringDB for Internet eXchange

• Location and contact information

• Who is there, both on fabric and for cross connect– useful for planning when building out or searching for peers

• IXP-DB under development which will sync up with PeeringDB

5

Tools to create router configuration

• Typically use Internet Routing Registry (IRR) data– Mostly communicate with RADB, which mirrors other registry

data– APNIC members can use APNIC RR-DB

• IRRToolSet– The first public tool set, current development unclear but

probably stable

• BGPQ3– Newer tool, currently actively developed– Creates filters so you will need to script using it

6

IRRToolSet

• https://github.com/irrtoolset/irrtoolset

• Can create router configurations from policy defined in Routing Policy Specification Language (RPSL)

• Uses autnum, as-set and route objects

• rtconfig – creates configuration file

• peval – queries IRR data

• Example policy in autnum objects – AS2764 & AS7575

7

BGPQ3

• https://github.com/snar/bgpq3

• Creates AS path or route filters based on IRR data• Supports a variety of formats– IOS (both “classic” and XR), JUNOS, JSON, BIRD – Can also DIY format

8

BGPQ3 Example (IOS-XR)

9

% bgpq3 -PXl prefixset-as38442 AS38442P- prefix listX- IOS-XRl- name of generated entry

no prefix-set prefixset-as38442prefix-set prefixset-as3844227.123.128.0/18,43.245.56.0/22,103.244.228.0/22,183.81.128.0/20end-set

Using communities for filters

• Set a community when you import a route from a customer or create a static (aggregate) route

• Use that community to control export to peers & transit

• Don’t allow peers or transits to set it though

• Now when you add a prefix on a router it will automatically get exported on other routers without updating their prefix lists

10

Peering is not just technical

• A personal relationship helps

• Support your local NOG (Network Operator Group) • If you expand to other markets try to attend their

NOG or Peering Forums

• For TL networks, this might include:– APRICOT – Asia Pacific– Peering Asia – Asia Pacific– MyNOG – Malaysia– SGNOG – Singapore– IDNOG – Indonesia– PhNOG – Philippines

11

Introduction to Route CollectorsWhat routes are available at the IXP?

What is a Route Collector?

• Usually a router or Unix system running BGP

• Gathers routing information from service provider routers at an IXP– Peers with each ISP using BGP

• Does not forward packets

• Does not announce any prefixes to ISPs

14

Purpose of a Route Collector

• To provide a public view of the Routing Information available at the IXP– Useful for existing members to check functionality of BGP filters– Useful for prospective members to check value of joining the IXP– Useful for the Internet Operations community for troubleshooting

purposes• E.g. www.traceroute.org

15

Route Collector at an IXP

Route Collector

R1

R3

R5SWITCH

R2 R4

16

Route Collector Requirements

• Router or Unix system running BGP– Minimal memory requirements – only holds IXP routes– Minimal packet forwarding requirements – doesn�t forward any

packets

• Peers eBGP with every IXP member– Accepts everything; Gives nothing– Uses a private ASN– Connects to IXP VLAN

• �Back-end� connection– Second Ethernet globally routed– Connection to IXP Website for public access

17

Route Collector Implementation

• Most IXPs now implement some form of Route Collector

• Benefits already mentioned• Great public relations tool

• Unsophisticated requirements– Just runs BGP

18

Introduction to Route Servers

What is a Route Server?

• Has all the features of a Route Collector

• But also:– Announces routes to participating IXP members according to their

routing policy definitions

• Implemented using the same specification as for a Route Collector

20

Features of a Route Server

• Helps scale routing for large IXPs

• Simplifies Routing Processes on ISP Routers• Optional participation

– Provided as service, is NOT mandatory

• Optionally uses Policy registered in IRR

21

Diagram of N-squared Peering Mesh

• For large IXPs (dozens for participants) maintaining a larger peering mesh becomes cumbersome and often too hard

22

Peering Mesh with Route Servers

• ISP routers peer with the Route Servers– Only need to have two eBGP sessions rather than N

RS RS

23

RS based Exchange Point Routing Flow

TRAFFIC FLOWROUTING INFORMATION FLOW

RS

24

Advantages of Using a Route Server

• Helps scale Routing for very large IXPs

• Separation of Routing and Forwarding• Simplify Routing Configuration Management on ISPs

routers

25

Disadvantages of using a Route Server• ISPs can lose direct policy control

– If RS is only peer, ISPs have no control over who their prefixes are distributed to• Some IXPs provide community based filtering option

• Completely dependent on 3rd party– Configuration, troubleshooting, etc…

26

Typical usage of a Route Server

• Route Servers may be provided as an OPTIONAL service– Most common at large IXPs (>50 participants)– Examples: LINX, HKIX, AMS-IX, etc

• ISPs peer:– Directly with significant peers– With Route Server for the rest

27

Things to think about...

• Would using a route server benefit you?– Avoids having to maintain a large number of eBGP peers– But can you afford to lose policy control? (An ISP not in control of

their routing policy is what?)

28

IXP Best Common Practices

Concept

• Some Service Providers attempt to cash in on the reputation of IXPs

• Market Internet transit services as �Internet Exchange Point�– �We are exchanging packets with other ISPs, so we are an

Internet Exchange Point!�– So-called Layer-3 Exchanges — really Internet Transit

Providers– Router used rather than a Switch– Most famous example: SingTel-IX

Competition

• Too many exchange points in one locale– Competing exchanges defeats the purpose– Los Angeles and Tokyo have multiple but it’s a rarity

• Becomes expensive for ISPs to connect to all of them

Rules and Restrictions

• IXPs try to compete with their membership– Offering services that ISPs would/do offer their customers

• IXPs run as a closed privileged club– E.g. Restrictive membership criteria (closed shop)

• IXPs providing access to end users rather than just Service Providers

• IXPs interfering with ISP business decisions e.g. Mandatory Multi-Lateral Peering

Technical Design Issues

• Interconnected IXPs– IXP in one location believes it should connect directly to the

IXP in another location– Who pays for the interconnect?– How is traffic metered?– Competes with the ISPs who already provide transit between

the two locations (who then refuse to join IX, harming the viability of the IX)

– IXP spanning multiple data centres in a city work ok (e.g. LINX)

Technical Design Issues

• ISPs bridge the IXP LAN back to their offices– �We are poor, we can’t afford a router�– Financial benefits of connecting to an IXP far outweigh the

cost of a router– In reality it allows the ISP to connect any devices to the IXP

LAN• with disastrous consequences for the security, integrity and reliability of

the IXP

Routing Design Issues

• Route Server Mandated– Mandatory multilateral peering is generally not welcomed– ISPs have no incentive to learn BGP– Therefore have no incentive to understand peering

relationships, peering policies,– Entirely dependent on operator of RS for troubleshooting,

configuration, reliability• RS can’t be run by committee!

• Route Server is mainly to help easy peering at IXPs but should not just rely on it– Should do bilateral peering as well

More Information

eXchange Point Examples

• AMS-IX, DE-CIX and LINX in Europe• Equinix IX, in every Equinix Data Centre• SIX in Seattle, Washington, USA• SGIX in Singapore• MyIX in Kuala Lumpur, Malaysia• BBIX, JPIX and JPNAP in Tokyo, Japan• HKIX in Hong Kong• IX Australia in Perth, Sydney, Melbourne, Brisbane………• All use Ethernet Switches

HKIX• Started and owned by CUHK since 1995– Neutral to various service providers

• Gradually opened up to all networks to connect– Served licensed ISPs only initially

• 8 full-time staff– Run by part-time staff initially

• Expanded to 4 different commercial data centers in 2016-17 while still maintaining neutrality

• 290+ networks connected• Peak traffic at 1.1+Tbps • Top 3 IXP in Asia Pacific now in terms of traffic volume• See: http://www.hkix.net

SGIX

• Not-for-profit / Independent legal entity / Membership-based

• Set up with the support of Singapore Government• With full-time staff from beginning• Went through a painful start-up process when potential

participants preferring to take wait-and-see approach– Persistency helped them get through the difficult period

• 100+ networks now• Peak traffic at 180+Gbps• Has presence in 7 commercial data centers• See: https://www.sgix.sg

MyIX

• Non-profit / Independent Legal Entity / Membership-based

• Set up with the support of Malaysian Government• Full-time staff• Governance by elected representatives of local Malaysian

ISPs• Multiple nodes in multiple cities• 40 Ordinary Members (NSP licensees; with voting right)

/ 30+ Associate Members (without voting right) / 20+ Members by Invitation (without voting right)

• See: http://myix.my

APIX

• A forum for IXPs to exchange experiences.

• 26 IXPs from 17 economies• See: http://apix.asia

More info about IXPs

• https://www.pch.net/resources/wiki/– Another excellent resource of IXP locations, papers, IXP

statistics, etc

• https://www.internetexchangemap.com/– Tele Geography: A collection of IXPs and interconnect points

for ISPs

• https://www.peeringdb.com/– Searchable database of Exchange Points, Networks &

Facilities

Acknowledgement:• Philip Smith & Barry Greene

• Cisco Systems

46

Thank you