Ruxcon Finding Needles in Haystacks (the size of countries)
-
Upload
packetloop -
Category
Technology
-
view
905 -
download
5
description
Transcript of Ruxcon Finding Needles in Haystacks (the size of countries)
![Page 1: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/1.jpg)
Michael Baker@cloudjunky
Ruxcon - October 2012
Finding Needles in Haystacks(the size of countries)
Sunday, 21 October 12
![Page 2: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/2.jpg)
Acknowledgements
David Turnbull @dsturnbull
Gerald Kaszuba @gakman
Packetpig Committers
Sunday, 21 October 12
![Page 3: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/3.jpg)
Two Rules
Sunday, 21 October 12
![Page 4: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/4.jpg)
The Landscape.
Sunday, 21 October 12
![Page 5: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/5.jpg)
Exhibit A
CVE-2011-3192 - “Apache Killer”
auxiliary/dos/http/apache_range_dos 2011-08-19 normal Apache Range header DoS (Apache Killer)
Snort 1:19825
/Range\s*\x3A\s*bytes=([\d\x2D]+\x2C){50}/Hsmi
/Range\s*\x3A\s*bytes=([\d\x2D]+[\x2C\s]*){50}/Hsmi
Sunday, 21 October 12
![Page 6: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/6.jpg)
Prevention Fails.
Sunday, 21 October 12
![Page 7: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/7.jpg)
Detection is the key.
Sunday, 21 October 12
![Page 8: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/8.jpg)
NSM - “focused on providing an intrusion analyst with the best possible information in the shortest amount of
time” - NSMWiki
Sunday, 21 October 12
![Page 9: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/9.jpg)
Network Security Monitoring
Advocates focus on detection and that prevention will fail.
Believes in inventoried and defensible networks.
Build entropy from alert (attack) information.
Provide analysts with accurate information as fast as possible.
Sunday, 21 October 12
![Page 10: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/10.jpg)
Tools collect.
Sunday, 21 October 12
![Page 11: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/11.jpg)
People analyze.
Sunday, 21 October 12
![Page 12: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/12.jpg)
Network Security Monitoring
Squil
Argus
Flowgrep
Snort and Suricata
Bro
Network Miner
Netwitness
Sunday, 21 October 12
![Page 13: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/13.jpg)
It’s all about Context.
Sunday, 21 October 12
![Page 14: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/14.jpg)
Context
Enriched information, not just IP Addresses.
Additional intelligence on attackers.
Allow you to perform detective work
What if? Branch analysis and exploring data.
Providing full fidelity and full context quickly.
Sunday, 21 October 12
![Page 15: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/15.jpg)
Full Packet Capture
Complete record of all network data.
Provides the highest fidelity to analysts.
Only way to really understand subtle, targeted attacks.
Play, pause and rewind your network.
No need to have a specific logging setup.
Sunday, 21 October 12
![Page 16: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/16.jpg)
NSM + FPC > % OPTIONS
Sunday, 21 October 12
![Page 17: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/17.jpg)
bit.ly/RdrI6M
Sunday, 21 October 12
![Page 18: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/18.jpg)
“The difficulty shifts from traffic collection to traffic analysis. If you can store hundreds of gigabytes of traffic
per day, how do you make sense of it?” - Richard Bejtlich
Sunday, 21 October 12
![Page 19: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/19.jpg)
Big Data is a collection of data sets so large and complex that it becomes difficult to process
using on-hand database management tools. The challenges include capture, curation, storage,
search, sharing, analysis, and visualization. - Wikipedia
Sunday, 21 October 12
![Page 20: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/20.jpg)
Big Data
Cloud - Elastic compute and Cheap Storage
Map Reduce - parallel computation
Pig, Hive - avoid writing M/R
NoSQL - Cassandra and Mongo
Sunday, 21 October 12
![Page 21: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/21.jpg)
Map Reduce
Sunday, 21 October 12
![Page 22: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/22.jpg)
Sunday, 21 October 12
![Page 23: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/23.jpg)
Big Data ScaleI want to ask a 2.5TB question
Process 2.5TB, 8 hours, 4 Compute units.
Process 2.5TB, 4 hours , 8 Compute units.
Process 2.5TB, 2 hours, 16 Compute units.
Process 2.5TB, 1 hour, 32 Compute units.
Process 2.5 TB, 30 minutes, 64 Compute units.
Process 2.5 TB , 15 minutes, 128 Compute units.
Scale my compute to answer my question.
Sunday, 21 October 12
![Page 24: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/24.jpg)
Big Data Scale
0
125
250
375
500
4 8 16 32 64 12815
3060
120
240
480
Complex Job (Approx 2.5TB)
Minutes
Sunday, 21 October 12
![Page 25: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/25.jpg)
HistoryGoogle Map Reduce Whitepaper (2004)
Google File System Whitepaper (2003)
Hadoop is an Apache Project for M/R (2007)
Hadoop File System is a distributed file system for Hadoop nodes (2007)
Pig is a data analysis language to ease the creation of Map / Reduce jobs that run on Hadoop Clusters (2008)
Sunday, 21 October 12
![Page 26: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/26.jpg)
@packetpig
@packetpig = Packets (FPC) + Pig
Pig uses a data flow language called Pig Latin.
Executes Map/Reduce Jobs over Hadoop Clusters.
Works identically on-premise or in the cloud (Amazon’s EMR)
Sunday, 21 October 12
![Page 27: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/27.jpg)
FeaturesFull access to IP packets at scale.
Threat Analysis (Snort)
Traffic Analysis.
Flow-based deep packet inspection.
Geo-Location
Passive OS Detection (p0f)
File Dissection
Sunday, 21 October 12
![Page 28: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/28.jpg)
Sunday, 21 October 12
![Page 29: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/29.jpg)
Finding Zero Days
Sunday, 21 October 12
![Page 30: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/30.jpg)
Worth a coffee JD?Motivation
Time window
Attacker
Attack type
Target
Obfuscated
Anonymised
Sunday, 21 October 12
![Page 31: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/31.jpg)
Attacker Information
Sunday, 21 October 12
![Page 32: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/32.jpg)
File Extraction
Sunday, 21 October 12
![Page 33: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/33.jpg)
Big DataSecurity Analytics
Sunday, 21 October 12
![Page 34: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/34.jpg)
Anscombe’s QuartetII IIII IIIIII IVIV
x y x y x y x y0.0 8.04 10.0 9.14 10.0 7.46 8.0 6.58
8.0 6.95 8.0 8.14 8.0 6.77 8.0 5.76
13.0 7.58 13.0 8.74 13.0 12.74 8.0 7.71
9.0 8.81 9.0 8.77 9.0 7.11 8.0 8.84
11.0 8.33 11.0 9.26 11.0 7.81 8.0 8.47
14.0 9.96 14.0 8.10 14.0 8.84 8.0 7.04
6.0 7.24 6.0 6.13 6.0 6.08 8.0 5.25
4.0 4.26 4.0 3.10 4.0 5.39 19.0 12.50
12.0 10.84 12.0 9.13 12.0 8.15 8.0 5.56
7.0 4.82 7.0 7.26 7.0 6.42 8.0 7.91
5.0 5.68 5.0 4.74 5.0 5.73 8.0 6.89
Source: http://en.wikipedia.org/wiki/Anscombe%27s_quartet
Sunday, 21 October 12
![Page 35: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/35.jpg)
Anscombe’s Quartet
Source: http://visual.ly/anscombes-quartet
Sunday, 21 October 12
![Page 36: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/36.jpg)
Big Data Security AnalyticsVisualization
Fidelity
Interaction
Outlier Detection
Attacker Profiling
Enrichment
Transform
Prediction and Probability
Intelligence sharing
Statistical Analysis
Feature Extraction
Machine Learning
Sunday, 21 October 12
![Page 37: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/37.jpg)
This is not SIEM.
Sunday, 21 October 12
![Page 38: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/38.jpg)
Not SIEMFull Fidelity
Explore and explain the data (evidence).
Play, Pause and Rewind.
Blink and you miss it technology.
No aggregation.
No parsers or complex integration.
Clear intelligence.
Sunday, 21 October 12
![Page 39: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/39.jpg)
Visualisation
Sunday, 21 October 12
![Page 40: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/40.jpg)
Full HDPlay, Pause, Rewind
Sunday, 21 October 12
![Page 41: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/41.jpg)
Outlier Detection
Sunday, 21 October 12
![Page 42: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/42.jpg)
Classi#cation
Sunday, 21 October 12
![Page 43: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/43.jpg)
Sunday, 21 October 12
![Page 44: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/44.jpg)
Novelty and Outliers
Sunday, 21 October 12
![Page 45: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/45.jpg)
Sunday, 21 October 12
![Page 46: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/46.jpg)
Entropy and Covert Channels
Sunday, 21 October 12
![Page 47: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/47.jpg)
Enrichment
Sunday, 21 October 12
![Page 48: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/48.jpg)
Geocoding
Sunday, 21 October 12
![Page 49: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/49.jpg)
TOR
Sunday, 21 October 12
![Page 50: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/50.jpg)
Torrent Triangulation
Sunday, 21 October 12
![Page 51: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/51.jpg)
Transformation
Sunday, 21 October 12
![Page 52: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/52.jpg)
Network Graphs and Relationships
Sunday, 21 October 12
![Page 53: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/53.jpg)
Intelligence and Metric Sharing
Sunday, 21 October 12
![Page 54: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/54.jpg)
Indicators of Compromise
OpenIOC and CyBOX
Open Indicators of Compromise (XML)
Host and Network Indicators of Compromise
Fork a github repository
Execute Packetpig scripts that find bad things and visualise them
Sunday, 21 October 12
![Page 55: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/55.jpg)
DNS and Malware
Sunday, 21 October 12
![Page 56: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/56.jpg)
dns$V2
dns$V1
3aw.com.aua1040.g.akamai.net
adcentre.com.auapm.com.au
apndigital.com.auclassifieds.fairfax.com.au
dsa.f2.com.au.edgesuite.netfairfaxevents.com.au
fairfaxsyndication.comgoodguides.com.auinvestsmart.com.au
m.smh.com.aumagic1278.com.ausmhshop.com.au
subscriptions.fairfax.com.autradingroom.com.autributes.smh.com.au
twitter.comwww.3aw.com.au
www.adcentre.com.auwww.facebook.com
www.fairfaxevents.com.auwww.fairfaxsyndication.com
www.fox.com.auwww.goodguides.com.auwww.investsmart.com.auwww.magic1278.com.au
www.northerndailyleader.com.auwww.portnews.com.au
www.qt.com.auwww.rsvp.com.au
www.smartedition.smh.com.auwww.smhshop.com.au
www.stayz.com.auwww.sunshinecoastdaily.com.au
www.theage.com.auwww.thechronicle.com.auwww.theherald.com.au
www.themorningbulletin.com.auwww.tradingpost.com.auwww.tradingroom.com.au
www.triplem.com.auwww.tweednews.com.au
www.watoday.com.auwww.weatherzone.com.au
2000 4000 6000 8000 10000
Sunday, 21 October 12
![Page 57: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/57.jpg)
Analytics orSurveillance
Sunday, 21 October 12
![Page 58: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/58.jpg)
bit.ly/TzcSq8
Sunday, 21 October 12
![Page 59: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/59.jpg)
@packetpig @packetloop
Questions?
Sunday, 21 October 12
![Page 60: Ruxcon Finding Needles in Haystacks (the size of countries)](https://reader033.fdocuments.in/reader033/viewer/2022052410/555215a7b4c905852b8b53bf/html5/thumbnails/60.jpg)
Thank you!http://blog.packetloop.com
Sunday, 21 October 12