Inter-Peer NOC Communication
-
Upload
henry-pittman -
Category
Documents
-
view
23 -
download
0
description
Transcript of Inter-Peer NOC Communication
![Page 2: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/2.jpg)
Scene Setting: Straw Poll
Who here in this room does peering?
![Page 3: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/3.jpg)
Scene Setting: Straw Poll
Who here in this room does peering? Have you ever had issues resolving
problems with your peerings?– Difficulties contacting peers, finding the
right contact, communication problems?
![Page 4: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/4.jpg)
Scene Setting: Straw Poll
Who here in this room does peering? Have you ever had issues resolving
problems with your peerings? Do you maintain a local db of contacts?
– Why? Issues with freshness of data?
![Page 5: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/5.jpg)
Scene Setting: Straw Poll
Who here in this room does peering? Have you ever had issues resolving
problems with your peerings? Do you maintain a local db of contacts? When a peer needs to talk to you,
where does their call/email arrive?– Main NOC contact? Dedicated peering
contact? “Customer Care”?
![Page 6: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/6.jpg)
Scene Setting: Straw Poll
Who here in this room does peering? Have you ever had issues resolving
problems with your peerings? Do you maintain a local db of contacts? When a peer needs to talk to you,
where does their call/email arrive? Some names have been changed to
protect the innocent… and guilty…
![Page 7: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/7.jpg)
Why do you go peering?
Long term money savings Less Transit Lower latency, better performance Traffic Control Diversity, Reliability Presence
…and so on…
![Page 8: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/8.jpg)
Where’s the problem?
Poor inter-peer communication seems to be common– Friendly IX operator called in to “mediate”
Communication hitting the wrong place– Customer NOCs– IX Operator– IP address maintainer (e.g. whois contact)
![Page 9: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/9.jpg)
Identifying the right contact
Sources of information:– Whois queries to databases– IXP-maintained NOC and Peering contact db– Internal databases– Third-party voluntary databases
• http://puck.nether.net/netops list• peeringdb.com
All above are vulnerable to information “rot”
![Page 10: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/10.jpg)
How to drive RIPEdb/RA, etc
Some really subtle differences in the implementations– RIPE expects “AS” before an AS number!
Which contacts are useful Which objects to look up
– Like the Peer ASN, not the Peer IP address! Why can’t ASN be logged in adjacency
changes on routers?– This seems to drive IP-based lookups
![Page 11: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/11.jpg)
Drive the Data Sources Properly!
Example: using WHOIS queries “Oh, I have an outage on WAIX, I’ll look
up the IP address”$ whois -h whois.arin.net 198.32.212.11|less
…
OrgName: Exchange Point Blocks
…
RTechHandle: WM110-ARIN
RTechName: Manning, Bill
RTechPhone: +1-310-322-8102
RTechEmail: [email protected]
![Page 12: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/12.jpg)
Bad Data Enters the System
“Okay, I’ll phone Bill Manning”– But all Bill did was give WAIX some v4 space– Bill doesn’t run WAIX, and isn’t an operational
contact for WAIX
So, Bill either ignores your voicemail, or tells you to call someone else
Whatever – it’s added delay, increased frustration – it’s how not to do it
![Page 13: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/13.jpg)
Driving Whois Properly
Always lookup the PEER ASN– Not the IP address!– It’s a BGP problem, we use ASNs in BGP$ whois -h whois.ra.net AS3856|less
aut-num: AS3856
as-name: UNSPECIFIED
descr: Packet Clearing House
www.pch.net
admin-c: Bill Woodcock
tech-c: Bill Woodcock
remarks: [email protected], +1 866 BGP PEER
![Page 14: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/14.jpg)
Driving Whois Properly
Always lookup the PEER ASN– Not the IP address!– It’s a BGP problem, we use ASNs in BGP$ whois -h whois.ra.net AS3856|less
aut-num: AS3856
as-name: UNSPECIFIED
descr: Packet Clearing House
www.pch.net
admin-c: Bill Woodcock
tech-c: Bill Woodcock
remarks: [email protected], +1 866 BGP PEER
![Page 15: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/15.jpg)
So you’ve found the contact
How do they respond to you?– Confusing recursive call trees?– Recalcitrant ticketing systems?– First-line NOC – “Is it switched on?”– “You’re not a customer, go away”
Once negotiated, peering is an engineering relationship– So backbone ops, not “customer care”
![Page 16: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/16.jpg)
Expectations of Peer Contacts
Choose your points of contact carefully Big problems with
– What’s peering/BGP/WAIX?– Are you a customer?– What’s your circuit ID?– Go away, you aren’t a customer
All serious no-no’s – be nice to your peers!
![Page 17: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/17.jpg)
PCH INOC-DBA Phones
PCH operate a “dial by ASN” NOC hotline system– They run the SIP registry/proxy– “Bring your own” SIP compliant phone
The idea is that it should get through to someone clueful– No call-trees, no music-on-hold
http://www.pch.net/inoc-dba/
![Page 18: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/18.jpg)
Suggested Role Contacts
Peering@– For setting up new peerings, changing existing
ones, no 24x7 expectation– Shouldn’t go to exclusively to sales@ ;-)
NOC@– Reaches your 24x7 NOC, which is either BGP
friendly and has enable, or knows when, how and where to escalate
Support@– Is generally your “customer-care”/call center
![Page 19: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/19.jpg)
Getting the message across
Okay, so you’ve made contact– Now, make your point
Provide the peer with useful information
– Start with the subject line– Be informative, who, when, what– Messages like “Help” and “Peering down”
aren’t helpful
![Page 20: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/20.jpg)
How not to do it…
Where? How does it affect me?– All detail buried in wordy message body
When? No TZ stamp! Help me handle my huge NOC inbox!
-----Original Message-----
From: Joe Schmoe <[email protected]>
Sent: Wednesday, January 25, 2006 5:41 PM
Subject: Maintenance Notification
Dear Peers,
…
![Page 21: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/21.jpg)
Example: Useful Subject Headers
AS7132’s preferred subject line format: <IX location> - <peer writing to/ASN> - <peer writing from/ASN> - <what is the issue> - <date of initial correspondence> - <time of initial message>
Example subject line: Equinix-Ashburn - RCN/6079 - SBC/7132 - new session
turn-up - 29- Mar-06 - 9:45 am ESTThanks to Ren Provo
![Page 22: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/22.jpg)
Look clueful
What does this say about your peer?– Don’t you think they look silly?
Run tools to help you answer these questions yourself– Netflow, MAC accounting, etc.
Subject: Traffic Drop
Dear Peer,
We suddenly noticed a 300Mb drop in traffic on our connection to the PIE-IX. Can you investigate, and help us find where the traffic has gone?
Regards,
…
![Page 23: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/23.jpg)
How to escalate
Check your equipment first Ask your peer - “What’s up?”
– Often you can resolve a problem bi-laterally Go to the IX only if you need to
– Not all IX operators can provide a 24x7 contact When to escalate a customer fault
– Don’t stonewall customer reports– Don’t point them to the IX operator– Co-ordinate directly with your peers
![Page 24: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/24.jpg)
How the IXP Op can help
Provide an up-to-date list of IX participants and their NOC/Peering contact information– Usually password protected
Help break comms deadlock– Help fix “dead ends”
Otherwise, they can only help with “physical” problems– “link down”, packet loss, broken cables, packet
corruption to all destinations connected to the IXP
![Page 25: Inter-Peer NOC Communication](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812df3550346895d93535c/html5/thumbnails/25.jpg)
In Summary
Keep your own information up to date– Whois db objects, third party dbs
Make sure your peering and NOC contacts are appropriate– No-one likes call-trees and holding
Find the right contacts at your peers Be nice to your peers!