From Threat Intelligence to Rap sheets (in just a … 1/Dragos...•Finding other domains sharing...
Transcript of From Threat Intelligence to Rap sheets (in just a … 1/Dragos...•Finding other domains sharing...
1© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
FROM THREAT
INTELLIGENCE TO RAP
SHEETS
(IN JUST A FEW STEPS)Dragos Comaneci, Principal Software EngineerAdrian Hada, Security Research Engineer
2© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
ABOUT
• Software Engineer, Ixia
• Loves to build, debug & understand
distributed systems
• Security Research Engineer, Ixia
• Spends most of his time around malware,
botnets and the like
3© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
AGENDA
• What’s a Rap Sheet?
• Threat Identification
• Storing and Interpreting Data
• Stats
4© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
DISCLAIMER
• Not product placement
• Our perspective on developing a threat intelligence system
5© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
WHAT’S A RAP SHEET?
• “a list kept by the police of all the times a
person has been arrested” (m-w.com)
• an official police document that lists the
crimes that a particular person has
committed (dictionary.cambridge.org)
6© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
WHAT’S A RAP SHEET?
• Expanding upon the
idea
• Track all malicious
IPs on the Internet
over time
7© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
WHAT’S A RAP SHEET?
• IP address or domain
• Proof of maliciousness
• 100% certainty
• No moral judgement
8© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
WHAT’S A RAP SHEET?
• Malware is dirty
• Exploit kits are dirty
• Bots exploiting vulnerabilities are dirty
• Phishing pages are dirty
• Spam is “clean”
• pr0n is “clean”
Examples
9© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. | 9© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
THREAT IDENTIFICATION (FINDING
THE BADDIES)
10© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
THREAT IDENTIFICATION
• Battery of AV products
• Threat intelligence feeds
• Detection threshold
• Lower chance of False Positives
Virus Scanning
11© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
THREAT IDENTIFICATION
• No execution
• Interesting properties/artefacts
• Imported/mentioned functions
• Sections
• Entropy
• Certificates
• Particular strings
• Children’s game using Mimikatz
• Educational app dumping passwords
• Other intel on binary
Static Analysis
12© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
THREAT IDENTIFICATION
• Cuckoo Sandbox
• Great project!
• VM/sandbox hardening is a must
• Cuckoo does some of this work for you
• VMs are easy to revert and reuse
• (alternatively) Execute on hardware – slow cleanup
• Analyze behavior on execution
• All sorts of honey
• Applications, documents, credentials
• Grab more Intel – dropped files, contacted IPs, URLs, etc
Dynamic Analysis
https://xkcd.com/694/
13© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
THREAT IDENTIFICATION
• Responsibility!
• Try not to:
• Spam others
• Gathering spam is also useful
• DoS others
• Brute force others
• Infect others
• Sometimes more easier said than done!
Dynamic Analysis
14© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
THREAT IDENTIFICATION
• Multiple honeypots
• Dionaea, Kippo/Cowrie, Glastopf and others
• Globetrotting
• Different continents
• Different countries
• Different provider sizes
• Hard to administer!
Honeypots
Internet
15© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
THREAT IDENTIFICATION
• Learning from the enemy
• Honeypot proxies
• Forward to real honeypot
• Easy to deploy/redeploy
• No dependency/OS issues
• Honeypot blacklisting
• One jar for many flies
• Dump to central repository
Honeypots
Internet
Honey courtesy of Jeff Geerling and http://cliparts.co/
16© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
THREAT IDENTIFICATION
• False positives
• Scanners – malicious or not?
• Indexing bots – real or fake?
• Only identify attacks
• Signatures for attacks
Honeypots
17© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
THREAT IDENTIFICATIONPhishing Detection
18© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
THREAT IDENTIFICATION
• Static “signatures” for larger targets
• Plenty of challenges
• Signature development
• No false positives
• Originals will always match
• Phish of phish of phish…
• Redo periodically
• Limited detection
• Easy to bypass
• Must look Facebook-y or Google-y, not exact clone
Phishing Detection
19© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
THREAT IDENTIFICATION
• Generic, machine learning-based approach
• Processes the HTML code
• Tries to classify correctly
• False positives likely
Phishing Detection
20© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
THREAT IDENTIFICATION
• Hostnames, domains and IP addresses
• Valuable information
• Important infrastructure services (whitelisting)
• Reoccurrences
• Mapping threat actors
Passive DNS
21© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. | 21© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
HOW DO WE MAKE SENSE OF ALL
THE DATA WE’RE COLLECTING?
22© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
Rap Sheet System
STORING AND INTERPRETING DATAOverview
URLs
Binaries
Honeypot Attacks
Rap Sheets
Tracking Info:
• SHAs
• FQDNs
• IPs
• URLs
23© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
STORING AND INTERPRETING DATA
• Real-time system
• As soon as we have enough information to build a Rap Sheet, we build & publish it
• Dataflow model
• Each node receives some input and produces some output. E.g..
• Nodes are connected to one another in a Topology
• Nodes may interact with other external services (Databases, Storage, Sandbox execution, etc.)
• Special nodes
• Only produce output (e.g. scanning threat intelligence feeds and extracting URLs for processing)
• Only receive input (e.g. storing the final information into the database)
Some facts
URLs Fetcher Fetched contents
24© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
STORING AND INTERPRETING DATA
• Each node does one small thing (microservices anyone ? )
• Easy to develop & test
• Easy to reuse
• Easy to reason about
• Nodes can be combined in different ways in a topology
• Nodes can be scaled individually
• New nodes can be easily integrated
• Each node can be updated individually
Why this model ?
25© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
STORING AND INTERPRETING DATA
• Blob data
• Fetched URL contents, packet captures, dropped files, sandbox analysis results, screenshots, etc.
• Cloud storage (currently AWS S3)
• Structured data
• NoSQL multi-model DB called ArangoDB
• Key/value
• Document
• Graph
• Schemaless
• JSON objects
• Easy to add new fields & information
• AQL, transactions, indexes, joins
Where do we store data ?
26© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
STORING AND INTERPRETING DATAHow do we store things?
FQDN IP
URLSHA
Resolves to
Fetches to
• Four main entities with
associated information
• Graph links between them with
more information
27© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
STORING AND INTERPRETING DATA
• Graph naturally maps to the underlying problem domain
• We can do interesting queries like:
• Finding all IPs that have served a certain malicious SHA
• Finding redirector domains (& URL shortening services)
• Finding other domains sharing the same NS/MX servers
(as well the the usual PassiveDNS type queries)
• Finding IPs & Domains which served SHAs which were dropped during dynamic analysis by SHAs
coming from a specific IP.
• Flexibility:
• Can easily tack on new information & entities to the Graph
Why do we store it this way?
28© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
STORING AND INTERPRETING DATA
• Tech
• Datacenter OS & Apache Mesos
• Simple deployment for lots of distributed services (Redis, ELK, RabbitMQ, ArangoDB, Storm, etc.)
• Containerize all the things and then run them using Marathon
• Apache Storm
• Principles
• Split up your components
• Split up your database
• Common and battle hardened infrastructure components
• Moving code to data instead of the other way around
Scaling it up
29© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. | 29© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
STATS
30© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
STATSActive Rap Sheet Age
011
17
23
29
35
41
47
53
59
65
71
77
83
89
95
10
11
07
11
311
91
25
13
11
37
14
31
49
15
51
61
16
71
73
17
91
85
19
11
97
20
32
09
21
52
21
22
72
33
23
92
45
25
12
57
26
32
70
Number of Rap Sheets still considered malicious by age IP Count by Age
31© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
STATSDelisted Rap Sheet Age
01
32
12
73
33
94
55
15
76
36
97
58
18
79
39
91
05
111
11
71
23
12
91
35
14
11
47
15
31
59
16
51
71
17
71
83
18
91
95
20
12
07
21
32
19
22
52
31
23
72
43
24
92
55
26
12
71
Time an IP Continues to Behave Maliciously IP Count by Age
32© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |
STATS
• Top 20 – 85%
• “Usual suspects” – US, China, Germany
• Surprising – Vietnam, Indonesia
Malicious IP Addresses by Country
US35%
CN10%DE
5%
FR5%
RU4%
VN3%
BR3%
IT2%
GB2%
PL2%
UA2%
NL2%
KR1%
HK1%
IN1%
TR1%
MX1%
ID1%
AU1%
OTHER17%
IP Count By Country
US CN DE FR RU VN BR IT GB PL
UA NL KR HK IN TR MX ID AU OTHER
33© 2016 IXIA AND/OR ITS AFFILIATES. ALL RIGHTS RESERVED. |