Orbit1000 Technology DiscussionNANOG -1-

Opnix Smart Routing Technology Overview

”There is more then one way to skin a cat…”

Opnix Smart Routing Technology Overview

”There is more then one way to skin a cat…”

Aaron D. BrittOpnix, Inc.

Orbit1000 Technology DiscussionNANOG -2-

Orbit1000 Technology Discussion Overview

• Orbit1000 CPE Overview• Probing Method in More detail• Orbit1000 CORE Overview• Things to Come…• Lets Review - Q & A

Orbit1000 Technology DiscussionNANOG -3-

Orbit1000 CPE High Level Architecture Subscriber

AS 100

Carrier BAS 200

Carrier CAS 300

20.20.20.130.30.30.1

10.10.10.2

20.20.20.230.30.30.2

IP Block Advertised - (24.10.0.0/16)

OSPFArea 0

A

BC

EBGP

EBGPEBGP

IBGPIBGP

Orbit 1000

OrbitAS 64701

24.10.4.1

LAN 24.10.1.1

Opnix CORE

ENCRYPTED

Orbit1000 Technology DiscussionNANOG -4-

Functions of the Orbit1000 CPE• Probe stuff• Receive BGP Feed and Set Routes• Communicate with the CORE

– Send Raw Probe Data– Receive Optimized Routes

Orbit1000 CPE

Discovery ProbesSet BGP Routes

QA Probes

InternetCustomerRouter(s)

CORE

EN

CR

YP

TE

D

Orbit1000 Technology DiscussionNANOG -5-

How we become one with the Packet • UDP Probes – Proactive Philosophy using patented ActiveScan

– Tried ICMP - routers drop ICMP despite what RFC says– We tried TCP – set off IDS systems all over the place– We tried the force - but none of us had enough

metaclorians.– We now use a UDP probe, though proprietary in nature,

very similar to that of a typical traceroute.– We found that during testing, routing policy set using UDP

Probe data is within 2% of the routing policy set using TCP probe data, but it doesn’t set off IDS systems!

Orbit1000 Technology DiscussionNANOG -6-

Probing Mechanism• Where do we probe?

– Prefix List based on prefixes important to each Customer• Top 500 Trafficked Sites/ News Groups etc…• Route Feed from Customer Routers• Traffic Flow Data (Netflow, Span Port <sniff sniff>)• Logs (Web, DNS etc…)• Capable of probing 110,000+ routes, but it doesn’t make

sense to (most of the time)– discovery.ignore and discovery.include lists.– ’Prefix + 1’ methodology, unless a more specific ip address is

specified in the configuration.• We probe multiple prefixes over multiple upstreams in parallel,

configurable amount – how much bandwidth do you want to spend on Probes?

Orbit1000 Technology DiscussionNANOG -7-

Metrics Gathered • OpScore (Algorithm based on the probe data weighted, and

calculated based on customer defined settings)– Latency– Unreliability

• Link Unreliability• Probe Closure• Packet Loss• Routing Loops

– Bad Hops – Layer 3 Hops– Carrier Preference

• Lowest score wins

Prefix 216.183.192.0/19 Over Carrier "B" Prefix 216.183.192.0/19 Over Carrier "C"

Actual Weight Result Actual Weight Result25 25% 6.25 75 25% 18.75

Actual Weight Result Actual Weight Result15 10.00% 1.5 20 10.00% 2

Actual Weight Result Actual Weight Result1 10.00% 0.1 0 10.00% 0

Actual Weight Result Actual Weight Result50 25.00% 12.5 25 25.00% 6.25

Actual Weight Result Actual Weight Result125 30.00% 37.5 50 30.00% 15

OpScore 57.85 OpScore 42.00

Layer 3 Hops (range 2 to 30)

Bad Hops (range 1 to 5)

Unreliability (Range 1 - 100)

Latency (5 to 300 ms)

Carrier Preferenc (Range 100 - 1) Carrier Preference (Range 100 - 1)

Layer 3 Hops (range 2 to 30)

Bad Hops (range 1 to 5)

Unreliability (Range 1 - 100)

Latency (5 to 300 ms)

Orbit1000 Technology DiscussionNANOG -8-

QA Process (Testing the Active Link)• UDP Based (Just like our Discovery Probes)• We QA everything!• We send the QA probe to a TTL based on where we think the

endpoint is based on our discovery data.• We check the latency and unreliability against the probe data

we used to set the route. • How many QA routes do we send, and how fast?

– The QA Limit is configurable like Carrier Limit in the Client Config – which means you control how many routes we can QA in parallel.

• QA happens much faster then Discovery.

Orbit1000 Technology DiscussionNANOG -9-

Orbit1000 CORE• 5 Pieces

– Balancer (Communicates w/CPE)– Optimizer (Crunches Numbers)– View (Keeps Latest and Greatest Views per CPE)– SQL dB (Stores Stuff)– Customer Portal (Looks stuff up)

View

Customer Portal

SQL dB

Balancer OptimizerCPE

Portal

CORE

Orbit1000 Technology DiscussionNANOG -10-

Data Access• Portal

– Access to Data, raw and graphical (Current and Historical)– All metrics and weights represented– Access to each CPE Client Config– RouteVision (Visualize over Multiple Paths)– Aggregate Summarizations

• SQL dB– Raw Data

• Transactional Data (Real Time)• Warehoused Data (Portal)• Archival Data

Orbit1000 Technology DiscussionNANOG -11-

Fault Tolerance Stuff…• If it goes up in smoke, the Customer router reverts back to standard

BGP.• Discovery Probes halt if the CPE loses the CORE connection, if keep-

alives fail within a period of time, product removes routes and “sleeps” until communication with the CORE is reestablished.

• Fault Tolerant reasoning behind storing CPE config on central dB• Heartbeat / fail over process between CPE’s• SNMP traps, early warning system (RAM, Hard Disk, CPU etc..)• Always working on additional MIB support

Orbit1000 Technology DiscussionNANOG -12-

Things to Come…• Probes to support Jumbo Frames (Adjustable Frame Size)• Dedicated Jitter Metrics• Black- hole and Routing Loop Discovery/reports via Website• TCP Slow Start Algorithm emulation• TCP and/or UDP probes (Pick your poison)• TCP Sniffing for Active Links (Monitor Actual Data – Replace QA)• Multicast Support• IPV6 Support• Additional MIB support• NEBS Compliant (just kidding)

Orbit1000 Technology DiscussionNANOG -13-

Contact Information

If you have any questions or would like to comment and/or critique this method of ‘Cat Skinning’ (I would love for some hecklers to drop me a line, with-out peer review no progress is possible) here is my contact info…

http://www.opnix.comaaron@opnix.com

Case Studies available today…• Tier 1 ISP• Fortune 5 Enterprise• Fortune 100 Financial Institution• Internet2/Abilene Deployment

Orbit1000 Technology DiscussionNANOG -14-

Layer 3 Hops vs latency (30 day Summary)

ttl: 3 latency: 0.020716ttl: 4 latency: 0.024832ttl: 5 latency: 0.033791ttl: 6 latency: 0.045662ttl: 7 latency: 0.055674ttl: 8 latency: 0.079405ttl: 9 latency: 0.109979ttl: 10 latency: 0.131937ttl: 11 latency: 0.141727ttl: 12 latency: 0.142373ttl: 13 latency: 0.143105ttl: 14 latency: 0.151558ttl: 15 latency: 0.177103ttl: 16 latency: 0.196629ttl: 17 latency: 0.216883ttl: 18 latency: 0.231439ttl: 19 latency: 0.244841ttl: 20 latency: 0.263682ttl: 21 latency: 0.268043

0

0.05

0.1

0.15

0.2

0.25

0.3

1 3 5 7 9 11 13 15 17 19

Series1

Orbit1000 Technology DiscussionNANOG -15-

Prefixes are how many hops away?

ttl: 3 # prefixes: 2047ttl: 4 # prefixes: 473ttl: 5 # prefixes: 660ttl: 6 # prefixes: 1621ttl: 7 # prefixes: 2726ttl: 8 # prefixes: 3601ttl: 9 # prefixes: 4340ttl: 10 # prefixes: 5527ttl: 11 # prefixes: 7831ttl: 12 # prefixes: 8761ttl: 13 # prefixes: 9111ttl: 14 # prefixes: 13756ttl: 15 # prefixes: 9506ttl: 16 # prefixes: 7743ttl: 17 # prefixes: 7174ttl: 18 # prefixes: 4679ttl: 19 # prefixes: 4321ttl: 20 # prefixes: 2881ttl: 21 # prefixes: 1339

0

2000

4000

6000

8000

10000

12000

14000

16000

1 3 5 7 9 11 13 15 17 19

Series1

Orbit1000 Technology DiscussionNANOG -16-

Other Questions to ask…

• Is there a direct correlation between Hops and Latency? Hop count seems anecdotal, yet the numbers are quite convincing…

• How accurate does UDP measurements compare with TCP measurements when talking about Latency, Packet Loss and Throughput?

• How much does Asymmetrical routing, play a part in the world of Sub optimal routing?

• With Netflow stats, on average it seems that Routers only forward packets to 10% or so of the Global Rib, yet our routing Tables are tenfold +. Seems we can do something here, I just don’t know what, yet…