Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap...
-
Upload
adam-power -
Category
Documents
-
view
221 -
download
1
Transcript of Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap...
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap Algorithms for Bitmap Algorithms for Counting Active Flows on Counting Active Flows on
HighHighSpeed LinksSpeed Links
Cristian Estan, George Varghese, Mike Fisk
Computer Science and Engineering Department,University of California, San Diego
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Why count flows?Why count flows?
• Detect port/IP scans• Identify DoS attacks• Estimate spreading rate
of a worm• Packet scheduling
Dave Plonka’s FlowScan
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Existing flow counting solutionsExisting flow counting solutions
ServerNetFlow data
Analysis
Traffic
reports
Network Operations Center
Router
Fast link
Memory
Network
Memory sizeMemory size
& bandwidth& bandwidth
NetworkNetwork
bandwidthbandwidth
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Motivating questionMotivating question
• Can we count flows at line speeds at the router?
– Wrong solution – counters
– Naïve solution – use hash tables (like NetFlow)
– Our approach – use bitmaps
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting algorithmsBitmap counting algorithms
• A family of algorithms that can be used as building blocks in various systems
• Algorithms can be adapted to application
• Low memory and per packet processing
• Generalize flows to distinct header patterns– Count flows or source addresses to detect attack– Count destination address+port pairs to detect scan
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Talk structureTalk structure
• Per packet processing for bitmap algorithms
• Computing flow count estimates from bitmaps
• Variance analysis of estimates
• Derived algorithms
• Related work
• Measurements
• Conclusions
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – direct bitmapBitmap counting – direct bitmap
HASH(green)=10001001
Set bits in the bitmap using hash of the flow ID of incoming packets
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – direct bitmapBitmap counting – direct bitmap
HASH(blue)=00100100
Different flows have different hash values
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – direct bitmapBitmap counting – direct bitmap
HASH(green)=10001001
Packets from the same flow always hash to the same bit
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – direct bitmapBitmap counting – direct bitmap
HASH(violet)=10010101
Collisions OK, estimates compensate for them
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – direct bitmapBitmap counting – direct bitmap
HASH(orange)=11110011
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – direct bitmapBitmap counting – direct bitmap
HASH(pink)=11100000
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – direct bitmapBitmap counting – direct bitmap
HASH(yellow)=01100011
As the bitmap fills up, estimates get inaccurate
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – direct bitmapBitmap counting – direct bitmap
Solution: use more bits
HASH(green)=10001001
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – direct bitmapBitmap counting – direct bitmap
Solution: use more bits
Problem: memory scales with the number of flows
HASH(blue)=00100100
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – virtual bitmapBitmap counting – virtual bitmap
Solution: a) store only a portion of the bitmap
b) multiply estimate by scaling factor
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – virtual bitmapBitmap counting – virtual bitmap
HASH(pink)=11100000
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – virtual bitmapBitmap counting – virtual bitmap
HASH(yellow)=01100011
Problem: estimate inaccurate when few flows active
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – multiple bmpsBitmap counting – multiple bmps
Solution: use many bitmaps, each accurate for a different range
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – multiple bmpsBitmap counting – multiple bmps
HASH(pink)=11100000
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – multiple bmpsBitmap counting – multiple bmps
HASH(yellow)=01100011
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – multiple bmpsBitmap counting – multiple bmps
Use this bitmap to estimate number of flows
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – multiple bmpsBitmap counting – multiple bmps
Use this bitmap to estimate number of flows
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – multires. bmpBitmap counting – multires. bmp
Problem: must update up to three bitmaps per packet
Solution: combine bitmaps into one
OR
OR
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
HASH(pink)=11100000
Bitmap counting – multires. bmpBitmap counting – multires. bmp
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting – multires. bmpBitmap counting – multires. bmp
HASH(yellow)=01100011
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Talk structureTalk structure
• Per packet processing for bitmap algorithms
• Computing flow count estimates from bitmaps
• Variance analysis of estimates
• Derived algorithms
• Related work
• Measurements
• Conclusions
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Basic estimatesBasic estimates
Direct bitmap
Virtual bitmap
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Multiresolution bitmap estimateMultiresolution bitmap estimate
Find most accurate Find most accurate componentcomponent
Estimate number of Estimate number of flows hashing to itflows hashing to it
Apply scaling factorApply scaling factor
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Talk structureTalk structure
• Per packet processing for bitmap algorithms
• Computing flow count estimates from bitmaps
• Variance analysis of estimates
• Derived algorithms
• Related work
• Measurements
• Conclusions
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Relative error in estimatesRelative error in estimates
Direct bitmap
Virtual bitmap
Multiresolution
bitmap
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Error of virtual bitmapError of virtual bitmap
Flow density (flows/bit)
Ave
rage
(re
lati
ve)
erro
r
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Memory requirementsMemory requirements
Direct bitmap < N / ln (Nε2+1)
Virtual bitmap 1.5441/ ε2
Multiresolution bitmap 0.9186 ln (Nε2) / ε2+ct.
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
100 million flows, error 1%100 million flows, error 1%
Hash table* 1.21 Gbytes
Direct bitmap 1.29 Mbytes
Virtual bitmap* 1.88 Kbytes
Multiresolution bitmap 10.33 Kbytes
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Talk structureTalk structure
• Per packet processing for bitmap algorithms
• Computing flow count estimates from bitmaps
• Variance analysis of estimates
• Derived algorithms
• Related work
• Measurements
• Conclusions
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Triggered bitmapTriggered bitmap
• Need multiple instances of counting algorithm (e.g. port scan detection)
• Many instances count few flows
• Triggered bitmap– Allocate small direct bitmap to new sources– If number of bits set exceeds trigger value, allocate
large multiresolution bitmap
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Talk structureTalk structure
• Per packet processing for bitmap algorithms
• Computing flow count estimates from bitmaps
• Variance analysis of estimates
• Derived algorithms
• Related work
• Measurements
• Conclusions
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Related workRelated work
• Flajolet, Martin (1985) probabilistic counting – Memory use similar to multiresolution bitmap
• Whang et al (1990) introduce direct bitmap• You, Chang (1996) use virtual bitmap• Chauduri, Motwani, Narasayya (1998)
– Counting flows without bias impossible from sampled data
• Duffield, Lund, Thorup (2002) – Accurate solutions based on counting TCP SYN flags
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Talk structureTalk structure
• Per packet processing for bitmap algorithms
• Computing flow count estimates from bitmaps
• Variance analysis of estimates
• Derived algorithms
• Related work
• Measurements
• Conclusions
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Multires. bmp. vs. prob. countingMultires. bmp. vs. prob. counting
Number of flows (log scale)
Ave
rage
(re
lati
ve)
erro
r
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Scan detection memory usageScan detection memory usage
Interval
length
Snort
(naïve)
Probabilistic
counting
Triggered
bitmap
12 seconds 1.94 M 2.42 M 0.37 M
600 seconds 49.60 M 22,34 M 5.59 M
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Talk structureTalk structure
• Per packet processing for bitmap algorithms
• Computing flow count estimates from bitmaps
• Variance analysis of estimates
• Derived algorithms
• Related work
• Measurements
• Conclusions
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
A family of counting algorithmsA family of counting algorithms
Setting Algorithm Applications
General counting Multiresolution bmp. Track infections
Narrow range Virtual bitmap Triggers (e.g. DoS)
Small counts common Triggered bitmap Port scans
Stationarity Adaptive bitmap Measurement
Add and delete Increment-decrement Scheduling
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Bitmap counting algorithmsBitmap counting algorithms
• A family of algorithms that can be used as building blocks in various systems
• Algorithms can be adapted to application
• Low memory and per packet processing
– With 2Kbytes error around 1%
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
The endThe end
Bitmap algorithms will be available at:
http://ial.ucsd.edu/bitmaps/
Any questions?
Acknowledgements: Vern Paxson, David Moore, Philippe Flajolet, Marianne Durand, Alex Snoeren, K Claffy, Stefan Savage, Florin Baboescu, NIST,NSF
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Adaptive bitmapAdaptive bitmap
• Virtual bitmap measures accurately number of flows if range known in advance
• Often number of flows does not change rapidly• Measurement repeated• Can use previous measurement to tune virtual
bitmap• Combine a large virtual bitmap with a small
multiresolution bitmap used for tuning
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Adaptive bitmap accuracyAdaptive bitmap accuracy
Number of flows (log scale)
Ave
rage
(re
lati
ve)
erro
r
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
With 2 kilobytes of memoryWith 2 kilobytes of memory
Adaptive bitmap
(min avg max)
Probabilistic counting
(min avg max)
Trace1 -4.4% 1.1% 4.7% -9.5% 2.8% 13.3%
Trace2 -1.9% 0.7% 2.0% -6.9% 2.8% 7.6%
Trace3 -1.8% 0.6% 1.8% 2.4% 10.2% 17.7%
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003
Increment-decrement algorithmsIncrement-decrement algorithms
• Active flow defined as flow with packets in queue
• Must support additions and deletions
• Replace bits of bitmap with counters– Increment when packet arrives– Decrement when packet leaves– Estimate number of flows based on zero counters