Can Drop but You Can’t Hide: persistent Estimation in High...
Transcript of Can Drop but You Can’t Hide: persistent Estimation in High...
![Page 1: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/1.jpg)
You Can Drop but You Can’t Hide: ‐persistent Spread Estimation in High‐speed Networks
Presenter: Prof. Shigang Chen
He Huang1, Yu-E Sun2, Shigang Chen3, Shaojie Tang4,
Kai Han5, Jing Yuan6, Wenjian Yang1
1School of Computer Science and Technology, Soochow University, China2School of Rail Transportation, Soochow University, China
3Department of Computer and Information of Science and Engineering, University of Florida, US4Naveen Jindal School of Management, University of Texas at Dallas, US
5School of Computer Science and Technology, University of Science and Technology of China, China6Department of Computer Science, University of Texas at Dallas, US
19th April, 2018IEEE INFOCOM 2018
![Page 2: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/2.jpg)
Traffic Measurement in High Speed Networks
2
Generalized Flow Size Measurement
Number of packets, number of bytes
Netflow
Generalized Flow Spread Measurement
Number of distinct elements in each flow, i.e. flow cardinality.
Scan detection, worm monitoring, proxy caching and content
access profiling, etc
![Page 3: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/3.jpg)
Flow size v.s. Flow spread
3
1000000 packets
Size = 1000000, Spread = 1
……
1 packet
1 packet
1 packet
Size = 100Spread = 100
![Page 4: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/4.jpg)
Persistent Spread
4
Stealthy DDoS attack, , , , ,
, , , , , Persistent element (source IP)
![Page 5: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/5.jpg)
Limitation of Prior Art
5
Stealthy DDoS attack, , , , ,
, , , , , Persistent element (source IP)
Limitation 1: Only count persistent elements that appear in all periods
Limitation 2: Assume transient elements appear in one period
![Page 6: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/6.jpg)
Problem Definition
6
We study a new problem called -persistent spreadestimation, which measures persist traffic elements ineach flow that appear during at least out of periods.
Other applications Identifying popular web files that are persistently accessed by
users over at least out of periods.
Profiling Internet access patterns
Monitoring scan activities
![Page 7: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/7.jpg)
Online Persistent Traffic Measurement
7
Extremely high line speed
On-chip memory shared by Routing
Packet scheduling
Access control
Quality of service
Packet inspection and classification
Intrusion detection
Traffic measurement
How to fit in an extremely tight memory space!
![Page 8: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/8.jpg)
Online Recoding
8
a bitmap for each flow f
0 0 0 0 0 0 0 0
0 1 2 3 4 5 6 71 11 1
![Page 9: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/9.jpg)
Offline Operation: Bitwise SUM
9
1 0 0 1 0 0 0 1
0 1 2 3 4 5 6 7
,
,
,
, ,
,
0 0 0 1 0 1 0 1,
1 0 0 0 0 0 0 1,
Bitwise SUM0 0 2 0 1 0 32
![Page 10: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/10.jpg)
Basic Idea
Known: , , fraction of counters whose valuesare
Unknown: , , number of elements that appearin out of measurement period.
Perdistent spread ∑ .
We derive the functional relationship between knownand unknown. , , provides T+1 equations to
solve for , .
0 0 2 0 1 0 32 V2 = 2 / 8
![Page 11: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/11.jpg)
Per DestinationFlow
Recording Many Flows with Virtual Bitmaps
11
One physical bitmap for all
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1
1
1 11 11 111 1
1 0 1 1 0 1 0 1 0 1 0 1 1 0 1 1 1 1 0 1 1 0 0 1
One virtual bitmap for each flow
![Page 12: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/12.jpg)
Virtual Bitmaps
12
Space saving
Implicit indexing
Noise in virtual bitmap
12
0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0
1
1 10 11 1 1
1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 1 1 0 0 1
![Page 13: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/13.jpg)
Experiment Results
Experiment setup: Dataset
One hour of data downloaded from CAIDA
38963 distinct flows, and 7179130 distinct elements
General setWe set 5 minutes as one measurement period.
Each study incoudes 8 measurement periods, i.e. T .
Memory ranges from 0.25MB ∼ 1MB.
13
![Page 14: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/14.jpg)
Experiment Results (cont.)
14
![Page 15: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/15.jpg)
Experiment Results (cont.)
15
![Page 16: Can Drop but You Can’t Hide: persistent Estimation in High ...home.ustc.edu.cn/~huang83/paper/INFOCOM18_slide.pdf · You Can Drop but You Can’t Hide: ‐persistent Spread Estimation](https://reader034.fdocuments.in/reader034/viewer/2022050609/5fafdaf088f158025828ac11/html5/thumbnails/16.jpg)
Base Station
Conclusion
A new traffic measurement problem that measuresnumber of persistent elements appearing in at leastout of predefined measurement periods.
A space-efficient solution for the problem
16