8 [France-IX] Technical (Simon) · 2018-09-24 · France-IX General Meeting September 2018 2...

Technical reportSimon MUYAL

France-IX General Meeting September 20181

Agenda

Technical key numbers

Paris & Marseille infrastructure upgrades

Zoom on outages

Securing route servers

Automating connection process


These achievements aretheresult ofthework donebythetechnical team

AlexandreArnaudAnastasiaBoulbabaMikaelPierrePierre-MaloThierryVittorio

Technical key numbers


New ports connected during last yearPeriod: From July 2017 to June 2018


Global1G +24

10G +62

100G +7

Global +93

Troubleshooting Tickets (NOC)


GlobalQ3-2017 54

Q4-2017 92

Q1-2018 75

Q2-2018 20

Global 241

Backbone infrastructure upgrade

Enabling Route server filtering

Other recurring requests• MAC address change• Portal credentials

Resp. time: 4.7/5Web portal: 4.2/5

Network AvailabilityNetwork availability : from july 2017 to june 2018


Marseille = 100%no outage

Paris = 99.995%4 outages (detailed later)

Paris infrastructure upgradesOptical infrastructureTelehouse 2Equinix PA6


Paris optical infrastructure upgrade8 Coriant Groove G30 deployed

Initial capacity deployed per link: 400G

PoPs upgraded:• Interxion PAR5• Telehouse 2• Interxion PAR2• Equinix PA6


SCHEMAavecliens400Gupgradé

400Glink

Paris optical infrastructure upgrade

Very easy to add 200G or 400G of capacity between 2 PoPs

Feedback after 12 months• No issue impacting members• Some python scripts to collect monitoring

information via APIs


Paris IP infrastructure upgrades: TH2Starting with core PoPs:

Interxion PAR5 : Done in July 2017

Telehouse2 : Done in November 2017

2 chassis with ~100 customers each one à 2 maintenancesHalf of our customers, half of our traffic…


Paris IP infrastructure upgrades: TH2


From Juniper EX9200 to Extreme SLX9850-8

Reminder

8 slots36 x 100G LC72 x 1G/10G LC

Paris IP infrastructure upgrades: TH2

Before migration: 2-3 months

• Intensive tests • Scripts to adapt configurations from Juniper to Extreme,

using our Information System• Power increase per rack• Interconnecting existing and new platforms to minimize

the downtime


Paris IP infrastructure upgrades: TH2During migration: 02:00 à 08:00am

Moving customers and monitoring in parallelMinimizing downtime per member : 5-10minutes in average

After migration: during some daysDouble checking using our tools:

• Observium: Global statistics, power/fan alerts, etc• Icinga: Customer IPs monitoring• BUM traffic sniffers: same amount of BUM traffic• QoS probes: same delays• BGP sessions with route servers: Always a good indicator


Paris IP infrastructure upgrades: TH2200 customers moved from Juniper EX9200 to Extreme SLX-9850


PA6 upgrade – July 2018

Increase of 100G requests

~ 20 members/30 ports to migrate

Extreme SLX 9850-8 deployed


BaiePA6aprèsmigration

Marseille infrastructure upgradesNew PoP MRS2Additional 100G LCRoute servers upgraded


Interxion MRS2 – May 2018

France-IX PoP in the new DC Interxion MRS2

Juniper EX9200 installed• 1G/10G/100G ports available

Interconnected to InterxionMRS1 with dual redundant path



Interxion MRS2 – May 2018Backbone capacity: 100G between Interxion MRS1 and MRS2, based on passive DWDM MUXes

Easy to evolve to 200G capacity

Possibility to install DWDM active equipment if traffic grows (400G)



BaieMRS2photo

Marseille infrastructure upgrades

In term of traffic, InterxionMRS1 is our 3rd PoP


BaieMRS2photo

1st2nd3rd

TH2PAR2

MRS1

among 12 PoPs

LCs installed to address 100G

customer requests @Interxion MRS1

Physical servers hosting route servers have been

replaced @Interxion MRS1 and

@Jaguar MRS01

Infrastructure upgrades: future worksSolution for edge PoPsInterconnecting Paris and Marseille


Edge : finding a solution for dense 100G PoPs

Reminder• Juniper EX9214 deployed in some edge PoPs• Not enough dense in terms of 100G ports and expensive

Solution found in 2017 with Brocade (SLX-9850 and SLX-9540)• But Extreme is not able to maintain pricing conditions obtained previously

Oct-Nov 2018 : Tender to find a box to replace progressively EX9214 when 100G needs increase


BaieMRS2photo

Interconnecting Paris and Marseille

Following a recent survey, members are asking for a connection between Paris and Marseille platforms

On the technical side, we will compare and select operators providing 100G waves between Paris and Marseille

We are currently working on how to provide this interconnection to our members

Available beginning of 2019France-IX General Meeting September 2018

22

BaieMRS2photo

Zoom on outages08/2018 – proxy ARP09/2018 – SLX platform


Outage : Proxy ARP

Proxy ARP configured on a customer portA member started replying to ”some” ARP requests, giving its MAC address in the reply…

A similar issue occurred 3 years ago

Our tools didn’t detect immediately the issue because the member replies only when the MAC address was not in its cache…


BaieMRS2photo

Outage : Proxy ARP: Detecting faster

In the quarantine VLAN, scan all the France-IX LAN IP range, and not only few IPs

Analyse BUM sniffers in real time• Ongoing work• Interesting not only for proxy ARP issue

Proactive: DAI: Dynamic ARP inspection• Analyse ARP traffic and filter ARP replies when they didn’t match

well known static entries (IP <-> MAC)• Available on Extreme SLX platform• Already tested in our lab, waiting for the vendor feedback


BaieMRS2photo

Outage : Extreme SLX platform instabilities

Extreme SLX-9850-8, platform stable since September 2017, 4 chassis deployed, connecting 250 customers

• 2 issues encountered over the last 12 months• February 2018 : Reload of one chassis at TH2: 9 minutes

• Combination of several issues• a lack of memory solved in the latest firmware installed• Now: Able to monitor memory and prevent this type of issue

• September 2018 : Reload of 2 100G LC during a logs collection: 8 minutes

• Command identified and documented on our side• Waiting feedback from vendor


BaieMRS2photo

Securing route serversApplying strict filtering based on ROA/RPKI and IRR


Securing route servers: Story

France-IX started securing route servers 18months agoBGP communities used to tag routes

• IRR not found• ROA invalid

Done using a combination of our DB, bird, some scripts and tools like bgpq3, NTT DB

Every member was able to filter by himself… but this is not enough to have a secure Internet Exchange...


51706:65012 = Prefix has ROA status: VALID51706:65022 = Prefix has ROA status: INVALID51706:65023 = Prefix has ROA status: UNKNOWN51706:65011 = Prefix is present in an AS's announced AS/AS-SET51706:65021 = Prefix is not present in an AS's announced AS/AS-SET

Securing route servers: Story

Sept 2017 : Following GM-2017, interesting discussion on tech-ML about applying strict filtering, based on IRR and ROA tags

Oct 2017 : Survey launched to have a better feedback, and not only few people expressing their opinion

Nov 2017: Survey results:• 110 members expressed their opinion!!• 73% in favour of a strict filtering based on ROA and IRR tags

Calendar for applying strict filtering:• Initially announced for January 2018• Applied in February 2018


Securing route servers: issues

• Technically, very simple!• “IRR not found” and “ROA invalid” BGP communities

were already applied on routes present in the RS…• Just a new rule in bird to filter these routes by default...

But...

• In nov 2017, 30% of members using RS had at least 1 “ROA invalid” or “IRR not found” route...

• Some announcements made on MLs


BaieMRS2photo

Securing route servers: issues

Members having 100% of their routes tagged as invalid:• Invalid AS-SET in peeringdb for example

Members having some of their routes filtered:• AS-SET not up-to-date• Some route objects missing in the IRR

It took some time to reach a situation where members updated their IRR objects


BaieMRS2photo

Securing route servers: Giving visibility

A dedicated looking glass to check route validityhttps://lg.franceix.net/welcome/RS1+RS2/ipv4

Documentation describinghow filtering is donehow we consider a route is invalidhttps://www.franceix.net/en/technical/france-ix-route-servers/

à Helping membersà Saving a lot of time for support and troubleshooting


Securing route servers: D Day

No main issue encountered

No traffic loss

A couple of members reacted to previous emails to update their route objects


BaieMRS2photo

Securing route servers: D Day

Paris7000 IPv4 routes filtered among 100 0004000 IPv6 routes filtered in Paris 28 000


BaieMRS2photo

Automating connection process


Automating connection process: DONE

Reminder: Our information system based on netbox

• Automating LOA• patch panel integration: Very long to check with DCs the

real status of a patch panel position...• Automating welcome mails

• Automating config checks


BaieMRS2photo

Automating connection process: Ongoing

50% of new ports delivered are related to upgrades

• Working on an easy way for members to start the upgrade process and save time

• Rewriting tools.franceix.net to provide additional information

• Survey: One of the points to improve• Technical information (IPs, ports, MACs, patchpannels, RS status

with ROA/IRR filtering)• Commercial view: services, billing, ...• Available beginning of 2019


BaieMRS2photo

THANK YOU FOR YOUR ATTENTION


[email protected]

+33 (0)1 70 61 97 72

twitter.com/ixpfranceix

facebook.com/ixpfranceix

youtube.com/user/TheFranceIX

8 [France-IX] Technical (Simon) · 2018-09-24 · France-IX General Meeting September 2018 2...

Documents

Transcript of 8 [France-IX] Technical (Simon) · 2018-09-24 · France-IX General Meeting September 2018 2...