NetFlow and flow-tools @ OARnet Mark Fullmer [email protected].

NetFlow and flow-tools @ OARnet

Mark Fullmer

[email protected]

Agenda

• OARnet - who we are, what we do.• Flow-tools summary - software we use to

collect and process NetFlow data.• NetFlow deployment at OARnet.• How we use NetFlow. Detailed examples

with flow-tools. Hopefully some tips that can be used or passed on to customers.

OARnet

• Internet access to about 100 of Ohio’s colleges and universities.

• Some commercial clients / co-lo services.• Internet2 / Abilene access to member

school’s in Ohio.• Sink over 1Gb/s of traffic from the Internet

during peak hours.

OARnet

• Multiple Transit providers, access to two of the NAPs, OC48 to Abilene.

• In the middle of a state wide network rebuild, moving from ATM circuits over leased lines to POS and GigE over DWDM using Cisco 15454’s.

Flow-tools

• Started back in June of ‘96 working from Darren Kerr’s fdg.c (a very simple example of displaying NetFlow v1 data). No commercial or open source NetFlow collectors available at the time.

• Released as an open source toolkit for working with NetFlow data from Cisco routers. At the time mostly used internally at Ohio State University, usually for network forensics.

Flow-tools

• Still available as open source, active development on and off as resources (time) are available. Keep working it as long as people are interested.

• Many other options available now for collectors and post processing both open source and commercial.

• A lot of sites use it, over 700 subscribers to the support mailing list.

NetFlow deployment

• Collector running FreeBSD and flow-capture at each POP (distributed collection).

• Sampled NetFlow data feeds (where software permits).

• Flow-capture stores the exports to disk with line compression at 5 minute intervals.

• Some reports run on the collectors, some at back at a central server (combination of distributed and centralized report generation).

NetFlow deployment

Flow collector.80GB RAID storage.

Core router

Aggregation routers.

OC48’s to other POPs.

NetFlow At Work

Traffic Engineering

• We have multiple transit providers with various contractual obligations.

• With provider T we must purchase a minimum of 300Mb/s / month.

• Provider Q recently reduced their prices to half of provider T.

• So we want to minimize our transit costs by using provider T for no more than 300Mb/s during peak hours until contract runs out (or they reduce their prices).

Traffic Engineering

Clearly above 300Mb/s on on outbound traffic, a littleover on inbound.

Traffic Engineering

• 170 prefixes that are announced to the internet.

• Prefix length does not necessarily correlate to traffic load. I.e. some smaller school’s with /16’s.

• So to influence provider T to send us less traffic to a prefix, pad the AS path.

Traffic Engineering

• Use NetFlow to determine which prefixes to pad.

• Select traffic that has an input interface of the provider circuit.

• Add local masks.• Scale the packets and octets by 100

(sampling rate is 1/100).• Summarize on destination IP & mask (zero

out the host bits).

Traffic Engineering

Collected flows(flow-capture).

Select a days data (flow-cat)

Filter on provider interface (flow-nfilter)

Fix masks (flow-mask)

Summarize by destination IP & mask (flow-report)

Format tabular data (flow-rptfmt)

Traffic Engineering

# ifIndex 46 is the POS interface to the Internetfilter-primitive CLMBO-R4-INTERNET type ifindex permit 46

# Match on traffic to the Internet by POS interface.filter-definition CLMBO-R4-INTERNET-OUT match output-interface CLMBO-R4-INTERNET

# Match on traffic from the Internet by POS interface.filter-definition CLMBO-R4-INTERNET-IN match input-interface CLMBO-R4-INTERNET

Flow-nfilter configuration:

Traffic Engineering

mask-definition AS600-ADV-ALL prefix 128.146.0.0/16 16 prefix 140.254.0.0/16 16 prefix 164.107.0.0/16 16 prefix 192.12.205.0/24 24 prefix 192.68.143.0/24 24 prefix 128.156.0.0/16 16 prefix 139.88.0.0/16 16 prefix 192.55.90.0/23 23 prefix 129.1.0.0/16 16 prefix 129.137.0.0/16 16. . .

Flow-mask configuration:

Traffic EngineeringFlow-report configuration:

stat-report CLMBO-R4-R8-FROM-INTERNET-BY-DESTINATION-PREFIX type ip-destination-address filter CLMBO-R4-INTERNET-IN ip-destination-address-format prefix-mask scale 100 output options +header,+xheader sort +octets fields -duration,+bps,+pps path @P/CLMBO-R4-FROM-INTERNET-BY-DESTINATION-PREFIX.txt output sort +octets fields -duration path |flow-rptfmt -Fip-destination-address,flows,octets,packets > \ @P/CLMBO-R4-FROM-INTERNET-BY-DESTINATION-PREFIX.fmt

Traffic EngineeringRun the report:

#!/bin/sh

FLOW_DATA=../CLMBO-R4/2004-11-08/*

flow-cat $FLOW_DATA | flow-report -s report.cfg -Sdaily-summaries \ -vP=daily-summaries/

Traffic EngineeringFlow-report output:

# --- ---- ---- Report Information --- --- ---# build-version: flow-tools 0.68p# name: CLMBO-R4-FROM-INTERNET-BY-DESTINATION-PREFIX# type: ip-destination-address# scale: 100# options: +header,+xheader# ip-dst-addr-type: prefix-mask# sort_field: +octets# fields: +key,+flows,+octets,+packets,+other# filter: CLMBO-R4-INTERNET-IN# records: 139# first-flow: 1099890083 Mon Nov 8 00:01:23 2004# last-flow: 1099976485 Tue Nov 9 00:01:25 2004# now: 1100221034 Thu Nov 11 19:57:14 2004## mode: streaming# compress: off# byte order: little# stream version: 3# export version: 5#

Traffic EngineeringFlow-report output: XXX

# recn: ip-destination-address*,flows,octets,packets164.107/16,1555282,227567453500,36933390064.247.64/18,2276007,145782119200,379959000129.137/16,678257,132932164100,179344100134.53/16,786497,110197887600,175669700198.30/16,523565,91210633100,126912900129.1/16,651195,77953466900,104152800130.101/16,389654,60952342600,87133000132.235/16,379746,55747741000,102050900140.254/16,328851,54698141100,65434300131.183/16,416366,53890273300,76920100128.146/16,511276,52719529000,92782100199.218/16,746850,52400585900,165697600130.108/16,308359,49463259900,66772600205.133/16,441714,44835214400,71746400206.244/16,268486,37445229500,50199700206.21/16,380376,35449280700,59676600…

Traffic EngineeringFlow-report-fmt output:

# ['./flow-rptfmt', '-Fip-destination-address,flows,octets,packets']ip-destination-address flows octets packets 164.107/16 1555282 227567453500 36933390064.247.64/18 2276007 145782119200 379959000129.137/16 678257 132932164100 179344100134.53/16 786497 110197887600 175669700198.30/16 523565 91210633100 126912900129.1/16 651195 77953466900 104152800130.101/16 389654 60952342600 87133000 132.235/16 379746 55747741000 102050900140.254/16 328851 54698141100 65434300 131.183/16 416366 53890273300 76920100 128.146/16 511276 52719529000 92782100 199.218/16 746850 52400585900 165697600130.108/16 308359 49463259900 66772600……….

Traffic EngineeringFlow-report-fmt output (top 10 percent totals):

# ['./flow-rptfmt', '-m10', '-p', '-Fip-destination-address,flows,octets,packets']ip-destination-address flows octets packets 164.107/16 10.906643 12.768776 12.74653964.247.64/18 15.960832 8.179813 13.113235129.137/16 4.756376 7.458804 6.189566 134.53/16 5.515425 6.183187 6.062754 198.30/16 3.671576 5.117815 4.380048 129.1/16 4.566600 4.373958 3.594546 130.101/16 2.732506 3.420027 3.007155 132.235/16 2.663024 3.127998 3.522005 140.254/16 2.306116 3.069105 2.258284 131.183/16 2.919828 3.023775 2.654685

Traffic Engineering

• A day long summary usually works for situations like this, but we may want to look at a time-series report over a day period (or longer).

• Gnuplot, or rrdtool, or gdchart (among others) can be used for this.

• Flow-rpt2rrd will convert the flow-report output to rrd format. For automated reports rrdtool works well since it bounds the storage requirements.

Traffic Engineering

#!/bin/sh FLOW_DATA=../CLMBO-R4/2004-11-08/* for name in $FLOW_DATA; do echo working...$name flow-report -s report.cfg -S5min-summaries \

-vP=5min-summaries < $namedone

for name in 5min-summaries/*; do echo working...$name ./flow-rpt2rrd -K as600.keys -p rrds < $namedone

Traffic Engineering

During peak hours moving traffic for 164.107/16elsewhere will drop inbound load by between 20 and 30Mb/s.

Traffic Engineering

• Minor changes to the report to get other results

• Outbound traffic by source prefix (busy sources in OARnet).

• Outbound traffic by destination prefix or AS (where are our sinks).

• Inbound traffic by source prefix (where do our sinks draw traffic from).

• Source / Destination pairs.

Traffic Engineering

stat-report CLMBO-R4-TO-INTERNET-BY-SOURCE-PREFIX type ip-source-address filter CLMBO-R4-INTERNET-OUT ip-source-address-format prefix-mask scale 100 output options +header,+xheader sort +octets fields -duration path @P/CLMBO-R4-TO-INTERNET-BY-SOURCE-PREFIX.txt

Traffic Engineering

stat-report CLMBO-R4-TO-INTERNET-BY-DESTINATION-AS type destination-as filter CLMBO-R4-INTERNET-OUT scale 100 output options +header,+xheader sort +octets fields -duration path @P/CLMBO-R4-TO-INTERNET-BY-DESTINATION-AS.txt

Traffic Engineering

# ['./flow-rptfmt', '-p', '-n', '-Fdestination-as,flows,octets,packets']destination-as flows octets packets SCRR-10796 3.769746 7.881135 4.664673 ATT-INTERNET3 4.636261 5.987038 4.844542 DNEO-OSP1 3.066557 3.309393 3.005959 CCINET-2 2.258435 2.896718 2.529854 ROGERS-AS 1.537620 2.664075 1.913102 CHARTER-NET-HKY-NC 2.369852 2.100978 2.175202 RR-CINCINNATI-ASN-01 0.732147 1.989616 1.190482 SCRR-11427 1.588737 1.646306 1.647938 VIDEOTRON-LTEE 0.735799 1.323654 1.078804 SCRR-11426 1.183881 1.289154 1.234044

Traffic Engineering

During peak hours moving traffic destined to AS 10796elsewhere will drop outbound load by between 20 and 30Mb/s.

RPF checks

• Provider Q wants to turn on strict RPF checks on our peerings.

• For a variety of reasons this won’t work for us (load distribution for one), but we do agree to implementing a traffic filter.

• There are places OARnet has not enforced RPF checks, mostly for historical reasons.

RPF checks

• So want to understand the impact of applying the traffic filter before applying it to prevent dropping valid customer traffic.

• Expect problems mostly to be multi-homed clients who may not be advertising all their address space to us.

• …Plus unexpected surprises.

RPF checks

Collected flows(flow-capture).

Select a days data (flow-cat)


Filter on announced address space

Format tabular data (flow-rptfmt)

Summarize results with flow-report

RPF checks

Flow-nfilter configuration:

#; OSUfilter-primitive AS600-ADV-ALL type ip-address-prefix default permit deny 128.146.0.0/16 deny 140.254.0.0/16 deny 164.107.0.0/16 deny 192.12.205.0/24 deny 192.68.143.0/24 deny 128.156.0.0/16 deny 139.88.0.0/16 deny 192.55.90.0/23 deny 129.1.0.0/16 deny 129.137.0.0/16...

RPF checks

Flow-report configuration:

stat-report CLMBO-R4-RPF-FAIL type ip-source-address filter CLMBO-R4-RPF-FAIL ip-source-address-format prefix-mask scale 100 output options +header,+xheader sort +octets fields -duration path |./flow-rptfmt stat-definition CLMBO-R4-RPF-FAIL report CLMBO-R4-RPF-FAIL

RPF checksResults

% FLOW_DATA=../clmbo-r4/2004-11-08/*% % flow-cat $FLOW_DATA | flow-report -s report.cfg -SCLMBO-R4-RPF-FAIL

ip-source-address flows octets packets198.234/16 24689 3478979000 489430066.144/15 22661 1422430800 3345200204.130.176/24 4328 398581200 737000 156.63/16 2387 279678500 387900 208.44/14 992 6527900 105700 65.112/12 845 4986800 88900 66.194.129.106/32 381 2150400 38400 53/8 91 1886400 39300 18/8 90 1737600 36200 45/8 89 1660800 34600 38/8 91 1622400 33800

Access Lists

• Same procedure can be used to test a firewall or access list before applying it to live traffic.

• Determine what the impact of a change will be before impacting customers.

Traffic Classification

• Abilene / Internet2 is a high speed research network that OARnet provides connectivity to for Ohio school’s.

• Only some school’s participate.• Of those school’s that participate they

can sponsor smaller school’s or other institutions for example a medical college.


• The participants all share the cost for an OC48 to Abilene. They want to know who’s using it both by school and group.

• School’s may have multiple networks.• Not all school’s have AS numbers.• School traffic is combination of prefixes.• Group traffic is combination of schools.


• Flow-tag adds a new 32 bit source tag and destination tag to the flows.

• Tags can be set based on criteria in the flow, such as source or destination prefix.

• Reports can then be run on the tagged traffic.


Collected flows(flow-capture)

Select a months data (flow-cat)

Add tags based on interface and prefix (flow-tag)

Graph tabular data (gnuplot)

Summarize results with flow-report


Traffic Classification## tag format## 0 7 15 23 31# 0000 0000 0000 0000 0000 0000 0000 0000 (32 bits)# RRRRRRRRRRRRRR TTTT NNNNNNNNNNNNNNNNNNN# | | | Site name# | | Site type# | Reserved

# ID Name#---------------------------------# 0x0001 OSU# 0x0002 CWRU# 0x0003 BGSU# 0x0004 UC# 0x0005 UAKRON# 0x0006 WRIGHT# 0x0007 KENT…

# ID Type#------------------------# 0x01 Participant# 0x02 SEGP# 0x03 Sponsored-Participant# 0x04 Gigapop # 0x05 MULTICAST

Traffic Classificationtag-action OHIO-GIGAPOP_DST type dst-prefix# OSU match 128.146/16 set-dst 0x010001 match 164.107/16 set-dst 0x010001 match 140.254/16 set-dst 0x010001 match 192.153.26/24 set-dst 0x010001 match 199.18.22.24/30 set-dst 0x010001 match 199.18.4.24/30 set-dst 0x010001# CHMCC match 204.152.48/24 set-dst 0x030014 match 204.152.49/24 set-dst 0x030014 match 205.142.196/24 set-dst 0x030014 match 205.142.197/24 set-dst 0x030014 match 205.142.198/24 set-dst 0x030014 match 205.142.199/24 set-dst 0x030014 match 199.18.107.220/30 set-dst 0x030014

Traffic Classificationstat-report TS-CLMBQ-R2-FROM-ABILENE-BY-TAG-GROUP type destination-tag tag-mask 0x00FF0000 0x00FF0000 filter CLMBQ-R2-ABILENE-IN scale 100 output options +header,+xheader sort +octets fields -duration path @P/CLMBQ-R2-FROM-ABILENE-BY-TAG-GROUP.%j.txt

stat-definition clmbq-r2-daily-summaries tag OHIO-GIGAPOP report TS-CLMBQ-R2-FROM-ABILENE-BY-TAG-GROUP report TS-CLMBQ-R2-FROM-ABILENE-BY-TAG-CUSTOMER report TS-CLMBQ-R2-TO-ABILENE-BY-TAG-GROUP report TS-CLMBQ-R2-TO-ABILENE-BY-TAG-CUSTOMER


• Groups of IP addresses - campus department traffic summaries.

• Individual IP addresses - Find busy hosts on a campus.

• Ports and protocols - find all the SMTP clients or hosts using file sharing services.


• We have the ability to provision dedicated ’s between many of the larger school’s.

• Cost money…Some clients want to know if they should be purchasing dedicated bandwidth between each other.

Enterprise/Campus proactive Security

• A lot of low hanging fruit.• Hosts with high packet/octet rates

to/from Internet.• Hosts that connect to a large number of

external sites.• Hosts that use many ports.• Usually not useful in our environment --

problems easy to find, hard to fix.

Enterprise/Campus security

• Usually use NetFlow to diagnose problems on the fly or historical events.

• Customer calls and wants to know why their T1 is full. Usually network scans or DoS events.

• A few weeks of historical data is very useful. If there is one compromised host there are usually others…


Hosts with known security issue(s)

Campus Network

Internet

NetFlow collector &Archive.

NetFlow v5 exports


Compromised hosts.

Campus Network

Internet


NetFlow v5 exports

Scan &compromise


Compromised hosts.1/5 used to attack.

Campus Network

Internet


NetFlow v5 exports

Remote trigger attack


• For some reason alerted to attack - proactive measures, local network slowdowns, victim network complaints, etc.

• Find and disable compromised host.• Use NetFlow archive to find suspicious traffic

from attacker.• Look back in history for other traffic from that

IP. Find other compromised hosts before they are used in future attacks.

• At least disable the attacker IP to campus.


• Good success rate at retroactively identifying worms and viruses.

• Not too smart, usually have same packet signature. Many 1 packet per flow from an infected host, or constant packet size, constant source or destination port.

• Can at least identify infected machines that are causing disruption to a campus.

• One would guess as more tools like NetFlow are deployed miscreants will make efforts to hide signatures.

Flow-tools

• A lot more is possible.• Flow-nfilter can filter on any fields in NetFlow

v1 - v8 flows.• Flow-report has about 80 built in reports.• Usually reports and graphs are generated

automatically.• On old PC with FreeBSD/Linux is enough to

get started.

Flow-tools

• Contributed software such as Dave Plonka’s FlowScan and Perl module http://net.doit.wisc.edu/~plonka

• Other information at http://www.splintered.net/sw/flow-tools

http://net.doit.wisc.edu/~plonka

http://www.splintered.net/sw/flow-tools

Thanks…

NetFlow and flow-tools @ OARnet Mark Fullmer [email protected].

Documents

Transcript of NetFlow and flow-tools @ OARnet Mark Fullmer [email protected].