Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates...

26
© 2013 Carnegie Mellon University Finding a Needle in a PCAP Emily Sarneso Flocon 2015

Transcript of Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates...

Page 1: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

© 2013 Carnegie Mellon University

Finding a Needle in a PCAP

Emily Sarneso

Flocon 2015

Page 2: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

2

Copyright 2014 Carnegie Mellon University.

This material is based upon work supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense.

References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University of its Software Engineering Institute.

NO WARRANTYTHIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

This material has been approved for public release and unlimited distribution except as restricted below.

This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at [email protected].

Carnegie Mellon®, CERT ® , CERT Coordination Center® and Flocon® are registered marks of Carnegie Mellon University.

DM-0001893

Page 3: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

3

GoalDescribe a full packet capture solution that can quickly and efficiently produce requested information.

Show analysis capabilities of YAF, super_mediator, and SiLK.

Demonstrate PCAP features in YAF.

Page 4: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

4

PCAP ChallengesVolume (4Gbps):• 1 Hour: 1.7TB• 1 Day: 40.8TB• 1 Week: 285.6TB• 1 Month: 1.1PBData Stored on Sensors• Separate from analysisIndexing:• Timestamp Files• BPF Filters• GUI tools• Splunk

Page 5: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

5

YAF PCAP FeaturesRolling PCAP dump

• Rotates files using time or size.• Creates meta file with flows contained in each PCAP file.

Index a PCAP File• Uses flow key hash and start time.

PCAP per flow• Creates a PCAP file for each flow.• Use with BPF filters.

Page 6: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

6

Gh0st Rat Investigation

Page 7: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

7

Gh0stRemote Access TrojanFree source codeEasy to modifyDistinctive Network Signature

SignatureUsually 5 BYTES

Compressed Length

4 BYTES

Uncompressed Length

4 BYTES

ZLIBHDR0x789C

2 BYTES

Data

Page 8: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

8

Method29,000 (15G) PCAP samplesUse YAF to index and produce flow, DPIYAF Signatures

Flow

Enhanced Flow (DPI)

PCAP

Page 9: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

9

Tool setup

Page 10: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

10

Initial Results

Page 11: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

11

YAF SignaturesNorman ASA 2012 Report identifies 85 Gh0st variants

download01.norman.no/documents/ThemanyfacesofGh0stRat.pdf

Page 12: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

12

Results with YAF Signatures

Page 13: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

13

Super_mediatorA very configurable IPFIX mediatorCollects every IPFIX information element YAF can exportMultiple exportersMultiple collectors (v.1.0)

YAF SUPERMEDIATOR

flowcap

FileStorage

SiLK

Page 14: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

14

Super_mediator configurationListing application label first allowed for quick binning by variant.

Super_mediator Results:• 227,833 Total Bi-flows• 60,816 Bi-flows Gh0st• 86,053 Unidentified

ApplicationHashStimemsDomainSipDipSportDportProtocolvlanintIflagsUflagsRiflagsRuflagsPkts,Rpkts

BytesRbytesDatabytesRdatabytesSmallpktsRsmallpktsLargepktsRlargepktsNonemptypktsRnonemptypktsMaxsizeRmaxsizeFirsteight

Page 15: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

15

Finding a Pattern

Page 16: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

16

Analysis Part 1Remove unwanted flows from unidentified flows:

• Remove flows with source/destination port 138,139.• Remove flows with initialTCPFlags = ‘R’• Remove flows with dataByteCount = 0

Find flows with pattern:• No more than 1 small packet (forward), 0 reverse• Non-empty packets = 1 or 2 (forward), 1 reverse• maxPacketSize = reverseMaxPacketSize• firstEightPacketDirection = 0x02

Results:• 44,468 bi-flows removed• 37,500 bi-flows with pattern• 4,085 bi-flows did not follow pattern

Page 17: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

17

Finding Gh0st Variants and Signatures

Page 18: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

18

Analysis Part 2Run unidentified PCAP files through YAF again and export first 100 bytes of payload

Page 19: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

19

ResultsIdentified several signature variants of Gh0stFound 55 new Gh0st variantsCreated YAF Application Label for Gh0st

• Correctly identifies 97% of Gh0st traffic.

Collected over 3,000 unique domain names• Correlated with Gh0st variants.

Page 20: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

20

Searching for Gh0st in DEFCON CTF PCAP

Page 21: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

21

DEFCON CTF PCAP DataGoal: Test new Gh0st application labelDefcon CTF PCAP Data

• 409 GB• Separated by team and

day

Page 22: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

22

Investigating “Gh0st” in DEFCON

Page 23: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

23

YafMeta2PcapInput:

• Large PCAP file or list of PCAP files• PCAP meta file created by YAF• Flow key hash and start time

Output• PCAP file with desired flow

Page 24: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

24

DEFCON AnalysisUsed YAF signatures to determine other flows with “DmdT” and “eliza”

“eliza” was a text-based space economy simulator challenge at CTF

80% of DmdT traffic went to last place team.

Page 25: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

25

Method Comparison

PCAP ->FLOW

yafMeta2Pcap

DeterminePCAP(S)

that contain

flow

MergePCAP files w/

mergecap

PCAP

Write a BPF filter that will return

session

SeparateFlows

TCPDUMP YAF

Page 26: Finding a Needle in a PCAP - SEI Digital Library. YAF PCAP Features. Rolling PCAP dump • Rotates files using time or size. • Creates meta file with flows contained in each PCAP

26

Questions?CERT NetSA tools website:tools.netsa.cert.org

Contact:[email protected]

[email protected]@cert.org