RIPE NCC Routing Information Service (RIS) › wp-content › uploads › ... · •Routing...

Oleg Muravskiy | ENOG 17 | October 2017

2017 Update

RIPE NCC Routing Information Service (RIS)

What is RIS?

Oleg Muravskiy | ENOG 14 | October 2017 3

What is RIS?

• Routing Information Service

• Worldwide network of BGP collectors

• Deployed at Internet Exchange Points

• Collects raw BGP data from peers

• Stores BGP updates and routing table dumps

• 15+ years of history

• Used by network operators and researchers every day

Oleg Muravskiy | ENOG 14 | October 2017

• 18 collectors and growing • 600+ peers • 150+ full-feed peers

4

Collector locations

Why RIS?Why are we doing this?

A bit of history


Why RIS?

• Original project was defined in RIPE-200 in 1999:

“In other words, it can be regarded as one integrated Looking-Glass for the entire Internet that includes history information”

• Looking glasses are instantaneous

• Routing problems are also instantaneous

• BGP history is recorded to track what is happening and what has happened

• Also to provide statistics and reporting on routing table metrics


Why the RIPE NCC RIS?

• RIPE NCC is a neutral body

• Experience running measurement platforms - Test Traffic Measurement project

- RIPE Atlas

• Supporting our own members - who are mainly network operators

• Supporting the community - researchers

- operators

RIS data accessWhat can you get?

And how do you get it?


Raw data!

• 15+ years of raw data (5.8 TB) available to download and analyse yourself :) - https://www.ripe.net/analyse/internet-measurements/

routing-information-service-ris/ris-raw-data

• Data stored in MRT (RFC6396) format

• Readable using BGPdump utility - open source, maintained by RIPE NCC

- https://bitbucket.org/ripencc/bgpdump

• …and by other tools

https://www.ripe.net/analyse/internet-measurements/routing-information-service-ris/ris-raw-data

https://www.ripe.net/analyse/internet-measurements/routing-information-service-ris/ris-raw-data

https://bitbucket.org/ripencc/bgpdump


Web interfaces and APIs

• Of course, if all we did was store the raw data, we’d just need a bunch of hard disks and an FTP server

• But you want to query all our lovely datasets!

• RIPEstat - https://stat.ripe.net/

- Our portal for everything you ever wanted to know!

https://stat.ripe.net/


RIPEstat

• RIPEstat (stat.ripe.net) is a web-based interface that provides everything you ever wanted to know about IP address space, Autonomous System Numbers (ASNs), and related information for hostnames and countries in one place


RIPEstat

• What can you search for? - ASN

- IPv4 address

- IPv4 prefix

- IPv6 address

- IPv6 prefix

- country


RIPEstat Examples


RIPEstat Examples: Selectel case


RIPEstat Examples

RIS data accessHow else can you get it


RIPEstat Data API

• All these queries are available through an API

• Actually, all those shiny web interfaces use the API anyway

• You can use it too!! Write your own scripts, etc.

• https://stat.ripe.net/docs/data_api

• There are also some extra API calls which are not yet visualised

https://stat.ripe.net/docs/data_api


RIPEstat Data API

• Remember this started because looking glasses are instantaneous?

• BGP State - https://stat.ripe.net/docs/data_api#BGPState

• This data call returns the state of BGP routes for a resource at a certain point in time, as observed by all the RIS collectors

• This is derived by applying a computation of state to the RIB dump (granularity=8h) that occurred exactly before that time, using the BGP updates observed between the RIB time and the query time

https://stat.ripe.net/docs/data_api#BGPState


RIPEstat Data API – BGP State – Selectel case

• Show me who and how was announcing this prefix at that time

• https://stat.ripe.net/data/bgp-state/data.json?resource=188.93.16.2&timestamp=2017-07-16T00:51:23Z

"data": { "bgp_state": [ { "source_id": "00-178.255.145.243", "path": [50304, 42708, 2854, 49505], "community": [], "target_prefix": "188.93.16.0/22" }, { "source_id": "00-193.0.0.56", "path": [3333, 1136, 24785, 24785, 24785, 24785, 20562, 2854, 49505], "community": [], "target_prefix": "188.93.16.0/22" }, ... ], "query_time": "2017-07-16T00:51:23", "resource": "188.93.16.0/22", "nr_routes": 44

https://stat.ripe.net/data/bgp-state/data.json?resource=188.93.16.2&timestamp=2017-07-16T00:51:23Z

https://stat.ripe.net/data/bgp-state/data.json?resource=188.93.16.2&timestamp=2017-07-16T00:51:23Z


Live stream demo

• Prototype!!

• Let’s see if it works

• http://stream-dev.ris.ripe.net/demo

• Live stream enables new applications - BGP hijack detection

- real time anomaly analysis

- live monitoring of your routes

http://stream-dev.ris.ripe.net/demo

What else can you do?Lots of analysis that this data allows


Prefix reachability studies

• https://labs.ripe.net/Members/emileaben/has-the-routability-of-longer-than-24-prefixes-changed

https://labs.ripe.net/Members/emileaben/has-the-routability-of-longer-than-24-prefixes-changed

https://labs.ripe.net/Members/emileaben/has-the-routability-of-longer-than-24-prefixes-changed


Prefix length visibility in IPv4

• https://labs.ripe.net/Members/dbayer/visibility-of-prefix-lengths

https://labs.ripe.net/Members/dbayer/visibility-of-prefix-lengths



Prefix length visibility in IPv6

• https://labs.ripe.net/Members/dbayer/visibility-of-prefix-lengths




BGP update propagation• https://labs.ripe.net/Members/vastur/the-shape-of-a-bgp-update

https://labs.ripe.net/Members/vastur/the-shape-of-a-bgp-update

RIS growthBecause the internet keeps growing


Collector historyCollector Location IXP Deployed RemovedRRC00 Amsterdam Multi-hop 1999 -RRC01 London LINX 2000 -RRC02 Paris SFINX 2001 2008RRC03 Amsterdam AMS-IX 2001 -RRC04 Geneva CIXP 2001 -RRC05 Vienna VIX 2001 -RRC06 Tokyo DIX-IE 2001 -RRC07 Stockholm Netnod 2002 -RRC08 San Jose MAE-West 2002 2004RRC09 Zurich TIX 2003 2004RRC10 Milan MIX 2003 -RRC11 New York NYIIX 2004 -RRC12 Frankfurt DE-CIX 2004 -RRC13 Moscow MSK-IX 2005 -RRC14 Palo Alto PAIX 2005 -RRC15 Sao Paulo PTT-Metro SP 2006 -RRC16 Miami NOTA 2008 -RRC18 Barcelona CATNIX 2015 -RRC17RRC19 Johannesburg NAPAfrica JB 2016 -RRC20 Zurich SwissIX 2015 -RRC21 Paris FranceIX 2015 -RRC22 Bucharest InterLAN coming soonRRC23 Singapore Equinix SG coming soon


Size of compressed raw BGP updates per monthgi

gaby

tes

0

10

20

30

40

50

60

7020

01.0

120

01.0

620

01.1

120

02.0

420

02.0

920

03.0

220

03.0

720

03.1

220

04.0

520

04.1

020

05.0

320

05.0

820

06.0

120

06.0

620

06.1

120

07.0

420

07.0

920

08.0

220

08.0

720

08.1

220

09.0

520

09.1

020

10.0

320

10.0

820

11.0

120

11.0

620

11.1

120

12.0

420

12.0

920

13.0

220

13.0

720

13.1

220

14.0

520

14.1

020

15.0

320

15.0

820

16.0

120

16.0

620

16.1

120

17.0

420

17.0

9

rrc00rrc01rrc02rrc03rrc04rrc05rrc06rrc07rrc08rrc09rrc10rrc11rrc12rrc13rrc14rrc15rrc16rrc18rrc19rrc20rrc21


Size of compressed BGP dumps per monthgi

gaby

tes

0

10

20

30

40

50

60

7020

01.0

120

01.0

620

01.1

120

02.0

420

02.0

920

03.0

220

03.0

720

03.1

220

04.0

520

04.1

020

05.0

320

05.0

820

06.0

120

06.0

620

06.1

120

07.0

420

07.0

920

08.0

220

08.0

720

08.1

220

09.0

520

09.1

020

10.0

320

10.0

820

11.0

120

11.0

620

11.1

120

12.0

420

12.0

920

13.0

220

13.0

720

13.1

220

14.0

520

14.1

020

15.0

320

15.0

820

16.0

120

16.0

620

16.1

120

17.0

420

17.0

9

rrc00rrc01rrc02rrc03rrc04rrc05rrc06rrc07rrc08rrc09rrc10rrc11rrc12rrc13rrc14rrc15rrc16rrc18rrc19rrc20rrc21


Number of IPv4 prefixes seen

0

100000

200000

300000

400000

500000

600000

700000

800000

900000

2015

-11

2015

-12

2016

-01

2016

-02

2016

-03

2016

-04

2016

-05

2016

-06

2016

-07

2016

-08

2016

-09

2016

-10

2016

-11

2016

-12

2017

-01

2017

-02

2017

-03

2017

-04

2017

-05

2017

-06

2017

-07

2017

-08

2017

-09

2017

-10/0/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30/31/32


Number of IPv6 prefixes seen

0

10000

20000

30000

40000

50000

2015

-11

2015

-12

2016

-01

2016

-02

2016

-03

2016

-04

2016

-05

2016

-06

2016

-07

2016

-08

2016

-09

2016

-10

2016

-11

2016

-12

2017

-01

2017

-02

2017

-03

2017

-04

2017

-05

2017

-06

2017

-07

2017

-08

2017

-09

2017

-10


Data growth

• More BGP routes - BGP table has grown from 60,000 to 600,000 routes

- more BGP updates

- larger RIB (table) dumps

• More RIS collectors

• More peers at each collector

• Non-linear growth curve ;)

RIS OperationsAs the system has evolved


Original architecture (1999)

• Diagram from RIPE-200 (original concept)

• Note “RIS Server” - singular!

• Also, the “database” - this becomes the hardest

part!!


“Classic” architecture (2003, 9 collectors)

James Aldridge . RIPE 44 , January 2003, Amsterdam . http://www.ripe.net/ris3

“RIS Classic” - OverviewBGPd

RawData

Periodic dumps ofPeriodic dumps ofRIB and UpdatesRIB and Updates

RawData

rsync

MySQLdbinsert

RawData

MySQLReplication

UI

rrcXXrrcXX

abcoudeabcoude halfweghalfweg

rsync

Collect & Schedule


“Classic” architecture (2003, 9 collectors)


Problems• Database insertion of data from 9 route collectors

on a single central machine is slow– Little headroom to allow for abnormal cases– Can sometimes take more than 24 hours to insert a

single day’s data– Little capacity to add more RRCs or full BGP feeds

• Limited attributes are stored in the database:– Only first 255 characters of AS Path stored– Other BGP attributes (communities, MEDs, etc.) ignored


“RISng” architecture (2003, 9 collectors)


RISng - OverviewBGPd

RawData

RawData

rsync

RawData

MySQLReplication

UI

rrcXXrrcXX

abcoudeabcoude halfweghalfweg

rsync

MySQL

dbinsert

MySQL

Replication

MySQLMySQL MySQLMySQL


Scaling the Database

• MySQL: splitting and sharding - 8 MySQL servers

- some collectors were so big they needed their own MySQL server!

• Data retention - database was only query-able for 3 months worth of data

- the references grew too large, that every 3 months we basically had to drop all the data, and let it start again

• Time for Big Data!


Big Data processing

• Apache Hadoop - An open-source software framework for distributed storage and

distributed processing of very large data sets on computer clusters built from commodity hardware

• Allows us to build a scalable storage and processing cluster - Attributes and aggregations for all historical data are available

• Currently over 150 servers in the cluster - Although the cluster is not only used for RIS

- Also used by RIPE Atlas and other projects


Big Data processing – components

• HDFS - distributed, replicated, cluster filesystem

• HBase - non-relational distributed database - large tables – billions of rows × millions of columns

• Map/Reduce - massive batch job processing

• Azkaban - “crontab” for Hadoop, with dependency tracking and more

• Kafka - Message Queue and stream processing


Scaling the collectors

• Quagga used as BGP collector

• Single-threaded - Not as scalable on modern multi-core CPUs

• Locks updates during table-dump process - Requires that dump completes before the hold timer expires, or BGP

session will drop

• Some data consistency issues - Sometimes updates are missing from the update dumps at the time of a

table dump

- This makes it difficult to accurately rebuild BGP state at a intermediate time, if updates are not reliable in-between

RIS operationsTime for a redesign

(and this is the current design!)


New architecture

• Multiple BGP daemons (ExaBGP) – at least 1 per core - lightweight daemon

- finally could saturate RRC server

• Message Queue - RabbitMQ ➠ Kafka

• Stream processing - raw updates files

- Looking Glass

• Batch processing - aggregations


Data collection


Back-end data distribution

How can you help?


How can you help?

• Peer with us!!! - AS12654

- http://www.ris.ripe.net/cgi-bin/peerreg.cgi

• Send us your routes - If you can send us your full

BGP table, please do

- If not, send us what you can!

- We will be recording them forever ;-)

CIXP, Geneva (RRC04)

DIX-IE, Tokyo (RRC06)

Netnod, Stockholm (RRC07)

MIX, Milan (RRC10)

NYIIX, New York City (RRC11)

MSK-IX, Moscow (RRC13)

PAIX, Palo Alto (RRC14)

PTTMetro, Sao Paulo (RRC15)

NOTA, Miami (RRC16)

Catnix, Barcelona (RRC18)

NAP Africa JB, Johannesburg (RRC19)

SwissIX, Zurich (RRC20)

France-IX, Paris (RRC21)

http://www.ris.ripe.net/cgi-bin/peerreg.cgi

http://www.ris.ripe.net/cgi-bin/peerreg.cgi


Interested in hosting an RRC?

[email protected] https://ripe.net/ris https://stat.ripe.net

https://ripe.net/ris

https://stat.ripe.net

RIPE NCC Routing Information Service (RIS) › wp-content › uploads › ... · •Routing...

Documents

Transcript of RIPE NCC Routing Information Service (RIS) › wp-content › uploads › ... · •Routing...