Sinkhole Routing

8/12/2019 Sinkhole Routing

1/83

Sink Holes

A Swiss Army Knife ISP Security

ToolVersion 1.5

Barry Raveendran Greene -- [email protected]

Danny McPherson -- [email protected]


2/83

Context

n ISP Security Real World Techniquesendeavor to share tools and techniquesthat our peers are using to enhance theirnetworks.

n Backscatter Traceback (NANOG 23)

n Security on the CPE Edge (NANOG 26)n Sink Hole (now NANOG 28)


3/83


4/83

Sink Hole Routers/Networks

n Sink Holes are the network equivalent of a

honey pot.

n BGP speaking router or workstation built to suck inand assist in analyzing attacks.

n Used to redirect attacks away from the customer working the attack on a router built to withstand the

attack.

n Used to monitor attack noise, scans, and other activity

(via the advertisement of default or unused IP space )


5/83


Target of

Attack

172.168.20.1 is attacked

172.168.20.0/24 targets network

Sink Ho le Netw ork


6/83


Target of

Attack



Router advertises172.168.20.1/32

Sink Ho le Netw ork


7/83


n Attack is pulled awayfrom customer and

aggregation router.n Can now do classification

ACLs, Flow Analysis,Sniffer Capture,

Traceback, etc.

n Objective is to minimizethe risk to the networkwhile investigating theattack incident.

Target of

Attack



Router advertises

172.168.20.1/32

S ink Hole Network


8/83


n Advertising Default from theSink Hole will pull down allsorts ofjunktraffic.

n Customer Traffic whencircuits flap

n Network Scans

n Failed Attacks

n Code Red/NIMDA

n Backscatter

n Can place tracking tools and

IDA in the Sink Hole networkto monitor the noise.

Customers



RouterAdvertises

Default

S ink Hole Network


9/83

Peer B

Peer A


IXP-W

IXP-E

Upstream

A

Upstream

A

UpstreamA

UpstreamA

UpstreamB

UpstreamB

Upstream

B

Upstream

B

POP

TargetTarget

NOCG

Sink Hole

Network

Sink Hole RouterReady to advertiseroutes and accept

traffic.

171.68.19.0/24

171.68.19.1


10/83

Scaling Sink Hole Routers/Networks

n Multiple Sinkholes can bedeployed within a network

n Combination of IGP with BGP

Triggern Regional deployment

n Major PoPs

n Functional deployment

n Peering pointsn Data Centers

n Note: Reporting more

complicated

Customers



S ink Hole Network


11/83

Why Sink Holes?

1. They work! Providers do use them intheir network.

2. More uses are being found throughexperience and individual innovation.

3. They take preparation.


12/83

Why call the technique a Sink Hole?

n Sink Hole is used to describe a technique thatdoes more than the individual tools weve had in

the past:n Black Hole Routers one router advertising dark IPspace.

n Tar Pits A section of a honey net or DMZ designedto slow down TCP based attacks to enable analysisand traceback

n Shunts redirecting traffic to one of the routersconnected interface.

n Honey Net a network designed to analyze andcapture penetrations.


13/83

Sink Hole Basics


14/83

The Basic Sink Hole

n Sinks Holes do not have to be complicated.

n Some large providers started their Sink Hole with a spareworkstation with free unix, Zebra, and TCPdump.

n Some GNU or MRTG graphing and you have a decentsink hole.

To ISP

Backbone

Sink Hole

Server

Advertise small

slices of Bogon and

Dark IP space


15/83

Expanding the Sink Hole

n Expand the Sink Hole with a dedicated router into a variety of

tools.n Pull the DOS/DDOS attack to the sink hole and forwards the

attack to the target router.

n Static ARP to the target router keeps the Sink HoleOperational Target Router can crash from the attack and thestatic ARP will keep the gateway forwarding traffic to the

ethernet switch.

To ISP Backbone

To ISP

Backbone

To ISP Backbone

Sink Hole

GatewayTarget Router

Sni f fers and

Analyzers

Static ARP to

Target Router


16/83

What to monitor in a Sink Hole?

n Scans on Dark IP.n Who is scoping out the network pre-attack planning.

n Scans on Bogons.n Worms, infected machines, and Bot creation

n Backscatter from Attacksn Who is getting attacked

n

Backscatter from Garbage traffic (RFC-1918leaks)n Which customers have leaking networks.


17/83

Monitoring Scan Rates

n Select /32 address from different block of your address

space. Advertise them out the Sink Holen Assign them to a workstation built to monitor and log

scans. ( Arbor Networks Dark IP Peakflow module is oneturn key commercial tool that can monitor scan rates.)

To ISP Backbone

To ISP

Backbone

To ISP Backbone

Sink Hole


Sni f fers and

Analyzers

Place various /32

Infrastructure

addresses here


18/83

Worm Detection & Reporting UI

Operator instantly

notified of Worminfection.

System automatically

generates a list ofinfected hosts forquarantine and

clean-up.


19/83

Monitoring Backscatter

n Advertise bogon blocks with no-export and an explicit

safety community (plus ISP egress filtering on the edge)n Static the bogon to a backscatter collector workstation

(as simple as TCPdump).

n Pulls in backscatter for that range allows monitoring.

To ISP Backbone

To ISP

Backbone

To ISP Backbone

Sink Hole


Sni f fers and

Analyzers

Capture

Backscatter Traffic

Advertise Bogons

with no-exportcommunity


20/83

Monitoring Backscatter

n Inferring Internet Denial-of-Service Activity

n http://www.caida.org/outreach/papers/2001/BackScatter/


21/83

Monitoring Spoof Ranges

n Attackers use ranges of valid (allocated blocks) and

invalid (bogon, martian, and RFC1918 blocks) spoofed IPaddresses.

n Extremely helpful to know the spoof ranges.

n Set up a classification filter on source addresses.

To ISP Backbone

To ISP

Backbone

To ISP Backbone

Sink Hole


Sni f fers and

Analyzers

Export ACL Logsto a syslog server

Classification ACL

with SourceAddress


22/83


Extended IP access list 120 (Compiled)permit tcp any any established (243252113 matches)

deny ip 0.0.0.0 1.255.255.255 any (825328 matches)









.

.

permit ip any any (600152250 matches)

Example: Jeff Nulls [[email protected]] Test


23/83


n Select /32 address from different block of your address space.Advertise them out the Sink Hole

n Assign them to a workstation built to monitor and log scans.n Home grown and commercial tools available to monitor scan rates (

Arbor Networks Dark IPApplication is one turn key commercial toolthat can monitor scan rates.)

To ISP Backbone

To ISP

Backbone

To ISP Backbone

Sink Hole


Sni f fers and

Analyzers

Place various /32

Infrastructure

addresses here


24/83

Safety Precautions

n Do not allow bogons to leak:n BGP no-export community

n Explicit Egress Prefix Policies (community,prefix, etc.)

n Do not allow traffic to escape the sinkhole:n Backscatter from a Sink Hole defeats the

function of a Sink Hole (egress ACL on theSink Hole router)


25/83

Black Hole Routers or Sink Holes?


26/83

Simple Sink Holes Internet Facing

n BCP is to advertise thewhole allocated CIDR

block out to the Internet.n Left over unallocated

Dark IP space gets pulledinto the advertising

router.

n The advertising routerbecomes a Sink Hole forgarbage packets.

Peer

Boarder

Aggregation

CPE

Internet

Backscatter Scanners Worms

Pulls ingarbagepackets.

Large CIDR Block Out

CustomersAllocated Block

CPE Router /w Default


27/83

ASIC Drops at Line Rate?

n Forwarding/FeatureASICs will drop packets

with no performanceimpact.

n Line Rate dropping will

not solve the problem

of garbage packetssaturating the link.

Peer

Boarder

Aggregation

CPE

Internet


Garbagepacketssaturate

link.





28/83

Backbone Router Injecting

Aggregates

n Some ISPs use theBackbone/core routers to injecttheir aggregates.

n Multiple Backbone injection pointsalleviates issues of link saturation,but exposes the loopbackaddresses (at least the way it isdone today).

n In a world of multiple Gig-Botsand Turbo worms, do you reallywant you backbone routersplaying the role of garbagecollectors?




Peer

Boarder

Aggregation

CPE

Internet


Garbagepackets

heads to the

backbonerouter

Backbone


29/83

Simple Sink Holes Customer

Facing

n Defaults on CPEdevices pull in

everything.n Default is the ultimate

packet vacuum

cleaner

n Danger to links duringtimes of securityduress.

Peer

Boarder

Aggregation

CPE

Internet

BackscatterScanners

Worms






30/83

Simple Sink Holes Impact Today

n In the past, this issue ofpulling down garbage

packets has not been a bigdeal.

n GigBots and Turbo Wormschange everything

n Even ASIC-based

forwarding platforms getimpacted from the RFC1812 overhead.

Peer

Boarder

Aggregation

CPE

Internet







31/83

Sink Holes Advertising Dark IP

n Move the CIDR Block Advertisements (or at least more-specifics ofthose advertisements) to Sink Holes.

n Does not impact BGP routing route origination can happenanywhere in the iBGP mesh (careful about MEDs and aggregates).

n Control where you drop the packet.

n Turns networks inherent behaviors into a security tool!

To ISP Backbone

To ISP

Backbone

To ISP Backbone

Sink Hole


Sni f fers and

Analyzers

Target router

receives thegarbage

Advertise CIDR

Blocks with Static

Lock-ups pointing

to the target router


32/83

Anycast Sink Holes to Scale

Anycast al lows garbage packet

load management and

dis t r ibut ion .

Core Backbone

RegionalNode

RegionalNode

Regional

Node

RegionalNode

Regional

Node

Regional

Node

ISPs ISPs ISPs

POPs

POPs

POPs

POPs

POPs

POPs


33/83

AnycastingSink Holes

Scaling Sink Holes on existinginfrastructure


34/83

Anycast and Security: Applications

nAnycast is a technique sucessfully used in thecommunity:

n DNS Servicesn Distributed Sink Holes

n Black Hole Routers - Dark IP Space Management (BGP

Lock-up static routes to Null0)

n Routing ConvergancenAnycast provides a tool to plug in Sink Holes

through out an existing network.


35/83

Anycast DNS Caches

Peer B

Peer AIXP-W

IXP-E

UpstreamA

UpstreamA

UpstreamA

UpstreamA

UpstreamB

UpstreamB

Upstream

B

Upstream

B

POP

CustomerCustomer

Primary DNS Servers

Sink Hole

Network

171.68.19.0/24

171.68.19.1 DNS CachingServer Cluster

SAFE - Architecture

DNS Caching

Server Cluster

DNS Caching

Server Cluster

DNS Caching

Server Cluster

DNS Secondary

Server Cluster

DNS SecondaryServer Cluster



36/83

Anycast DNS Caches

Peer B

Peer AIXP-W

IXP-E

UpstreamA

UpstreamA

UpstreamA

UpstreamA

UpstreamB

UpstreamB

Upstream

B

Upstream

B

POP

CustomerCustomer

Primary DNS Servers

Sink HoleNetwork

171.68.19.0/24


SAFE - Architecture

DNS Caching

Server Cluster

DNS Caching

Server Cluster

DNS Caching

Server Cluster

DNS Secondary

Server Cluster



DNS Forwarded to

the closed Caching

Cluster


37/83

Anycast Sink Holes

Peer B

Peer AIXP-W

IXP-E

UpstreamA

UpstreamA

UpstreamA

UpstreamA

UpstreamB

UpstreamB

Upstream

B

Upstream

B

POP

CustomerCustomer

Primary DNS Servers

171.68.19.0/24

171.68.19.1

Services Network

Remote TriggeredSink Hole

DNS Forwarded to

the closed Caching

Cluster


Remote Triggered

Sink Hole

Remote Triggered

Sink Hole



Remote Triggered

Sink Hole

Remote Triggered

Sink Hole


38/83

Anycast What is needed?

n Two IP Addresses: One address for management & Oneaddress for anycasting.

Router

Eth0192.168.1.2/30

Lo010.0.0.1/32

Eth0

192.168.2.2/30

Eth0192.168.3.2/30

Lo0

10.0.0.1/32

Lo010.0.0.1/32

Server Instance A

Server Instance B

Server Instance C

BGP IGPRedistribution

Destination Mask Next-Hop Dist

0.0.0.0 /0 127.0.0.1 0192.168.1.0 /30 192.168.1.1 0192.168.2.0 /30 192.168.2.1 0

192.168.3.0 /30 192.168.3.1 010.0.0.1 /32 192.168.1.2 110.0.0.1 /32 192.168.2.2 1

10.0.0.1 /32 192.168.3.2 1

Round-robin load balancing

Courtesy of Bill Woodcock Packet Clearing House (www.pch..net)


39/83

Anycast and Sink Holes

n Sink Holes are designed to pull in attacks.

n Optimal placement in the network requires

mindful integration and can have substantialimpact on network performance and availability

nA single Sink Hole might require major re-

architecting of the networknAnycast Sink Holes provide a means to distribute

the load throughout the network.


40/83

Anycast Sink Holes Example

Core Backbone

RegionalNode

RegionalNode

Regional

Node

RegionalNode

Regional

Node

Regional

Node

Temp late Backbon e with

Regional Centers

ISPs ISPs ISPs

POPs

POPs

POPs

POPs

POPs

POPs


41/83

Anycast Sink Hole Placement

Core Backbone

RegionalNode

RegionalNode

Regional

Node

RegionalNode

Regional

Node

Regional

Node

Place Sink Holes in each of th e

Region al Nodes

ISPs ISPs ISPs

POPs

POPs

POPs

POPs

POPs

POPs


42/83

Anycast Sink Holes

nAnycast Sink Holes are in their earlystages.

n Placement and control of the triggerrouters are the two interesting challenges.

n These challenges will dissolve as more

operational experience is gained.


43/83

Using Sink Holes to Protect

Infrastructure Point to Point Links


44/83

Protecting the Backbone Point to Point

Addresses

n Do you really need to reach the Backbonerouters Point to Point Address from any

router other than a directly connectedneighbor?

198.0.2.1 198.0.2.2BK-02-A BK-02-B


45/83


Addresses

n What could break?n Network protocols are either loopback (BGP, NTP,

etc.) or adjacent (OSPF, IS-IS, EIGRP).

n NOC can Ping the Loopback (alhough some tools suchas HP OV may have issues).

n Traceroutes reply with the correct address in thereply. Reachability of the source is not required.

198.0.2.1 198.0.2.2BK-02-A BK-02-B

BGP, NTP BGP, NTP

OSPF, ISIS, EIGRP OSPF, ISIS, EIGRP


46/83


Addresses

n What have people done in the past:

n ACLs Long term ACL management problems.

n RFC 1918 Works against the theme of the RFC

Traceroute still replies with RFC 1918 source address.

n Does not protect against a reflection attack.

192.168.2.1 192.168.2.2BK-02-A BK-02-B


47/83


Addresses

n Move the Point to Point Address blocks to IGPbased Sink Holes.n All packets to these addresses will be pulled into the

Sink Hole.n People who could find targets with traceroute cannot

now hit the router with an attack based on thatintelligence.

n Protects against internal and reflection based attacks.

BK-02-A BK-02-B

Sink Hole Module

Packet P-t-P

infrastructure address.

Packet P-t-P

infrastructure address.

198.0.2.1 198.0.2.2


48/83

Not Perfect Just Another Hurdle.

n Will not work with the routers on the border.

n By default, C (Connected) prefixes override all BGPinjected prefixes from the Sink Hole (you want this to

happen).n Basic security principle increment layers of security

there is never a perfect solution just additionalhurdles the more hurdles the better.

BK-02-B

Sink Hole Module

BK-02-A 198.0.2.1 198.0.2.2 BK-02-C198.0.2.6198.0.2.5

Internet

Dest = 198.0.2.2

Dest = 198.0.2.5


49/83


Addresses

Peer B

Peer AIXP-W

IXP-E

UpstreamA

UpstreamA

UpstreamA

UpstreamA

UpstreamB

UpstreamB

Upstream

B

Upstream

B

POP

CustomerCustomer

Primary DNS Servers

Sink HoleModule

171.68.19.0/24


SAFE - Architecture

Traffic destinedBackbone Links getsucked to Sink Hole

Anycast SinkHole Module




Anycast Sink

Hole Module

Anycast Sink

Hole Module

DOS Attack to

Backbone Interface


50/83

What if I do an ISP Edge ACL?

nAnti-Spoof and Anti-Infrastructure ACLs areencouraged on the edge. But .

n

Need to be everywhere to achieved desiredeffect including the customer edge (this isbeyond the BCP 38 requirements).

BK-02-B

Sink Hole Module

BK-02-A 198.0.2.1 198.0.2.2 BK-02-C198.0.2.6198.0.2.5

Internet

Dest = 198.0.2.2

Dest = 198.0.2.5

SRC = 198.0.2.5 | DEST = Customer

ReflectionAttack

InfrastructureACL


51/83

What if I do an ISP Edge ACL?

nAnti-Spoof and Anti-Infrastructure ACLs canbe combined with Sink Holing theInfrastructure Blocks.

n Remember it is all about adding hurdles.

BK-02-B

Sink Hole Module

BK-02-A 198.0.2.1 198.0.2.2 BK-02-C198.0.2.6198.0.2.5

Internet

Dest = 198.0.2.2

Dest = 198.0.2.5

SRC = 198.0.2.5 | DEST = Customer

ReflectionAttack

InfrastructureACL


52/83

Sink Holes and Turbo Worms

Are you ready for the next one?


53/83

The SQL Slammer Worm:

30 Minutes After Release

- Infections doubled every 8.5 seconds- Spread 100X faster than Code Red

- At peak, scanned 55 million hosts per second.


54/83


55/83

Expect Turbo Worms from All

Directions!

SQL

ISPs Backbone

InternalNetwork

DMZ Sink Holedetects TurboWorm that got

inside.

Sink Holes atvarious security

layers.


56/83

Turbo Worms - Conclusion

n The nature of the threat dictates that youneed to prepare before it happens.

n 30 minutes just enough time to react withwhat you have.n Remember the post-Slammer analysis

Slammers search algorithms were broken

n Sink Holes are one tool that has proventheir value especially with wormmitigation (after containment).


57/83

Know Your Network -- Be Prepared!


58/83

Questions?


59/83

Addendum - Materials


60/83

Sinkholes - Addendum

Construction


61/83

Sinkhole Router

Target of

Attack

Sniffer/Analyser

Neflow/SyslogCollector

Flow ofMgmt Data

SinkholeRouter

AnalysisSegment

MonitoringLink andInterface


62/83

Guidelines

n No IGP on Sinkhole

n iBGP

nPeering sessions via Management InterfacenSinkhole is a RRc

n Monitoring Interface to data-plane only

n Routes injected into IGP by router servicing theMonitoring Link


63/83

TestNet Address Allocation

Sinkhole Diversion Addresses192.0.2.8 -> balance

ANYCAST Sinkhole Address192.0.2.254

Monitor Link addresses

NOTE: provision these addresses in allSinkholes

192.0.2.4/30

All iBGP routers for Drop to NULL0

Purpose

192.0.2.1/32

Address Block


64/83

Sinkhole Router - Routing

Analyser

NeflowCollector

Advertise IGP LSAd.e.f.0/28

Not AddressedNo Routing

Statics192.0.2.8/32 ->192.0.2.6192.0.2.254/32 -> 192.0.2.6

NOTE: 192.0.2.4/30 is reusedat each Sinkhole

Static & iBGP192.0.2.1/32 -> NULL0192.0.2.254/32 ->NULL0

192.0.2.8/32 ->

192.0.2.5/30

192.0.2.6/30

d.e.f.1/29

d.e.f.2/29

d.e.f.3/29 d.e.f.4/29

Advertise IGP LSAs

192.0.2.8/32192.0.2.254/32

iBGPd.e.f.2 RRc of d.e.f.1d.e.f.1 NH=self


65/83

BGP Triggers for Sinkholes -

Addendum

Configuration


66/83

Trigger Routers Config

router bgp 100

.

redistribute static route-map static-to-bgp

.

!route-map static-to-bgp permit 10

description Std Redirect For Edge Drop

description - Use Static Route with Tag of 66

match tag 66

set origin igp

set next-hop 192.0.2.1

set community NO-EXPORT

!


67/83


!

route-map static-to-bgp permit 20

description Redirect For Sinkhole NULL0Drop

description - Use Static Route with Tag of 67match tag 67

set origin igp


set community NO-EXPORT 67:67

!!


68/83


!


description Redirect For Sinkhole Analysis


match tag 68set origin igp



!!


69/83


!


description Redirect For ANYCAST Sinkhole





!!


70/83


!


description Redirect For ANYCAST Sinkhole Analysis





!



71/83

Sinkhole Triggers

! Drop all traffic at edge of network

ip route 172.168.20.1 255.255.255.255 null0 tag 66

!

! Redirect victim traffic to Sinkhole

ip route 172.168.20.1 255.255.255.255 null0 tag 67!

! Redirect victim traffic to Sinkhole for Analysis

ip route 172.168.20.1 255.255.255.255 null0 tag 68


72/83


73/83

Sinkhole Router Config

router bgp 100

.

Neighbor peer-group INTERNAL

neighbor INTERNAL route-map Redirect-to-Sinkhole inneighbor INTERNAL remote-as 100

neighbor d.e.f.1 peer-group INTERNAL

!

route-map Redirect-to-sinkhole permit 10

description - Send to Router's NULL0 Interface

match community 67:67set ip next-hop 192.0.2.1

!


74/83



description - Send to Router's Analyser Intf

match community 68:68

set ip next-hop 192.0.2.8

!


75/83



description ANYCAST drop



!


76/83



description Anycast Analysis



!

Route-map Redirect-to-sinkhole permit 100


77/83

Sinkhole Router Routing

! For Std drop

ip route 192.0.2.1 255.255.255.255 null0

!

! For Analysisip route 192.0.2.8 255.255.255.255 interface FA0/0

!

! Bogus ARP for 192.0.2.8 to stop ARP request

ip arp 192.0.2.8 00.00.0c.99.99.99 arpa

!

! For ANYCAST Sinkhole Servicesip route 192.0.2.254 255.255.255.255 null0


78/83


n No Default static route in Sinkhole.

nSinkhole must not loop traffic back out

Management Interface.nTelnet access via router servicing theSinkholes Management Segment.


79/83


n No Default static route in Sinkhole.

nSinkhole must not loop traffic back out

Management Interface.nTelnet access via router servicing theSinkholes Management Segment.


80/83

Sinkhole Router

Sniffer/Analyser

Neflow/SyslogCollector

Flow ofMgmt Data

SinkholeRouter

AnalysisSegment

RedirectedTraffic


81/83

Sinkhole Analysis Services

n Local Netflow Collector and Analyser

n Local Syslog Server

nAnalyser remotely controlled

nI.e. VNC or Telnet


82/83

Results / Benefits

n Traffic pulled from Victim

n Control collateral damage

n iBGP Triggered

nAllows attack flow analysis


83/83

In-Depth Analysis

n Be careful: you must contain any attacktraffic, do not become a victim as well

nOutbound filtering: do not let sever connectback out at will

nOutbound filter ACE hits (and IP logs) willprovide additional information

Sinkhole Routing

Documents

Transcript of Sinkhole Routing