VAST: Visibility Across Space and Time Architecture &...

VAST: Visibility Across Space and TimeArchitecture & Usage

Matthias [email protected]

BroConAugust 19, 2014

http://matthias.vallentin.net

mailto:[email protected]

2 / 27

3 / 27

4 / 27

Outline

1. Introduction: VAST

2. ArchitectureOverviewExample Workflow: QueryData ModelImplementation

3. Using VAST

4. Demo

5 / 27

VAST: Visibility Across Space and Time

VASTA unified platform for network forensics

GoalsI Interactivity

I Sub-second response timesI Iterative query refinement

I ScalabilityI Scale with data & number of nodesI Sustain high & continuous input rates

I Strong and rich typingI High-level types and operationsI Type safety

VAST

Queries

10.0.0.1 10.0.0.254 53/udp10.0.0.2 10.0.0.254 80/tcp

Data

6 / 27

VAST & Bro

BroI Generates rich-typed logs representing summary of activity→ How to process these huge piles of logs?I Fine-grained events exist during runtime only→ Make ephemeral events persistent?

VAST: Visibility Across Space and TimeI Visibility across Space

I Unified data model: same expressiveness as BroI Combine host-based and network-based activity

I Visibility across TimeI Historical queries: retrieve data from the pastI Live queries: get notified when new data matches query

7 / 27

VAST & Big Data AnalyticsMapReduce (Hadoop)Batch-oriented processing: full scan of data+ Expressive: no restriction on algorithms- Speed & Interactivity: full scan for each query

In-memory Cluster Computing (Spark)Load full data set into memory and then run query+ Speed & Interactivity: fast on arbitrary queries over working set- Thrashing when working set too large

Distributed Indexing (VAST)Distributed building and querying of bitmap indexes+ Fast: only access space-efficient indexes+ Caching of index hits enables iterative analyses- Reduced computational model (e.g., no joins in query language)

8 / 27

Outline



3. Using VAST

4. Demo

8 / 27

High-Level Architecture of VAST

ImportI Unified data modelI Sources generate events

ArchiveI Stores raw data as eventsI Compressed chunks & segments

IndexI Secondary indexes into archiveI Horizontally partitioned

ExportI Interactive query consoleI JSON/Bro output

Export

IndexArchive

10.0.0.1 10.0.0.254 53/udp10.0.0.2 10.0.0.254 80/tcp

Import

9 / 27

Query LanguageBoolean Expressions

I Conjunctions &&I Disjunctions ||I Negations !I Predicates

I LHS op RHSI (expr)

ExamplesI A && B || !(C && D)I orig_h == 10.0.0.1 && &time < now - 2hI &type == "conn" || :string +] "foo"I duration > 60s && service == "tcp"

LHS: ExtractorsI &typeI &timeI x.y.z.argI :type

Relational OperatorsI <, <=, ==, >=, >I in, ni, [+, +]I !in, !ni, [-, -]I ~, !~

RHS: ValueI T, FI +42, 1337, 3.14I "foo"I 10.0.0.0/8I 80/tcp, 53/?I {1, 2, 3}

10 / 27

Outline



3. Using VAST

4. Demo

10 / 27

Query

client

1. Send query string to search2. Receive query actor3. Extract results from query

search

1. Parse and validate query string2. Spawn dedicated query3. Forward query to index

query

1. Receive hits from index2. Ask archive for segments3. Extract events, check candidates4. Send results to client

Client 11 / 27

Query

client1. Send query string to search

2. Receive query actor3. Extract results from query

search

1. Parse and validate query string2. Spawn dedicated query3. Forward query to index

query


Client

Search

11 / 27

Query



search1. Parse and validate query string

2. Spawn dedicated query3. Forward query to index

query


Client

Search

src == 10.0.0.1&&

port == 53/udp

Index

Partitions

Indexers

11 / 27

Query



search1. Parse and validate query string2. Spawn dedicated query

3. Forward query to index

query


Client

Search

src == 10.0.0.1&&

port == 53/udp

Index

Partitions

IndexersQuery

11 / 27

Query

client1. Send query string to search2. Receive query actor

3. Extract results from query

search1. Parse and validate query string2. Spawn dedicated query

3. Forward query to index

query


Client

Search

src == 10.0.0.1&&

port == 53/udp

Index

Partitions

IndexersQuery

11 / 27

Query



search1. Parse and validate query string2. Spawn dedicated query3. Forward query to index

query


Client

Search

Index

Partitions

IndexersQuery

src == 10.0.0.1&&

port == 53/udp

11 / 27

Query




query


Client

Search

Index

Partitions

IndexersQuery

port == 53/udpsrc == 10.0.0.1

11 / 27

Query




query


Client

Search

Index

Partitions

IndexersQuery

11 / 27

Query




query1. Receive hits from index

2. Ask archive for segments3. Extract events, check candidates4. Send results to client

Client

Search

Index

Partitions

IndexersQuery

11 / 27

Query




query1. Receive hits from index

2. Ask archive for segments3. Extract events, check candidates4. Send results to client

Client

Search

Index

Partitions

IndexersQuery

Archive

11 / 27

Query




query1. Receive hits from index2. Ask archive for segments

3. Extract events, check candidates4. Send results to client

Client

Search

Index

Partitions

IndexersQuery

Archive

11 / 27

Query




query1. Receive hits from index2. Ask archive for segments3. Extract events, check candidates

4. Send results to client

Client

Search

Index

Partitions

IndexersQuery

Archive

11 / 27

Query

client1. Send query string to search2. Receive query actor3. Extract results from query


query1. Receive hits from index2. Ask archive for segments3. Extract events, check candidates4. Send results to client

Client

Search

Index

Partitions

IndexersQuery

Archive

11 / 27

Outline



3. Using VAST

4. Demo

11 / 27

VAST Architecture

Export

IndexArchive

10.0.0.1 10.0.0.254 53/udp10.0.0.2 10.0.0.254 80/tcp

Import

12 / 27

Data RepresentationTerminology

I Data: C++ structures (e.g., 64ull)I Type: interpretation of data (e.g., count)I Value: data + typeI Event: value + meta data

I Type with a unique name (e.g., conn)I Meta data

I A timestampI A unique ID i where i ∈ [1, 264 − 1)

I Schema: collection of event typesI Chunk: serialized & compressed events

I Meta data: schema + time range + IDsI Fixed number of events, variable size

I Segment: sequence of chunksI Meta data: union of chunk meta dataI Fixed size, variable number of chunks

ID TIME “foo” 3.14 7 ms

META

META

TYPE

Event

Chunk

Segment

13 / 27

Types: Interpretation of DataTYPE

record

vector set

table

KEY VALUE

TYPETYPE

field 1

TYPE

field n

TYPE

…

container types

basic types

compound types

recursive types

bool

int

count

double

time range

time point

string

regex

address

subnet

port

none

14 / 27

VAST Architecture

Export

IndexArchive

10.0.0.1 10.0.0.254 53/udp10.0.0.2 10.0.0.254 80/tcp

Import

15 / 27

Index Hits: Sets of Events

Bitvector: sets of eventsI Query result ≡ set of event IDs from [1, 264 − 1)→ Model as bit vector: [4, 7, 8] = 0000100110 · · ·

Bitstream: encoded append-only sequence of bitsI EWAH (no patents unlike WAH, PLWAH, COMPAX)I Compact, space-efficient representationI Bitwise operations do not require decoding

Bitmap: maps values to bitstreamsI push_back(T x): append value x of type TI lookup(T x, Op ◦): get bitstream for x under ◦

0

264 � 1

.00001100001.

=

2

1

2

0

0

1

3

0

0

0

1

1

0

0

0

1

0

0

0

1

0

1

0

1

0

0

0

0

0

0

0

0

0

0

1

B1 B2 B3B0

Data Bitmap

16 / 27

Composing Results via Bitwise OperationsCombining Predicates

I Query Q = X ∧Y ∧ ZI x = 1.2.3.4 ∧ y < 42 ∧ z ∈ ”foo”

I Bitmap index lookup yields X → B1, Y → B2, and Z → B3I Result R = B1 & B2 & B3

& &

B1 B3

=

B2 R

17 / 27

Outline



3. Using VAST

4. Demo

17 / 27

Actor Model

Actor: unit of sequential executionI Message: typed tuple 〈T0, . . . , Tn〉 3 Tn

I Behavior: partial function over Tn

I Mailbox: FIFO with typed messagesI Can send messages to other actorsI Can spawn new actorsI Can monitor each actors

BenefitsI Modular, high-level componentsI Robust SW design: no locks, no data racesI Network-transparent deploymentI Powerful concurrency model

18 / 27

CAF: C++ Actor Frameworklibcaf

I Native implementation of the actor modelI Strongly typed actors available → protocol checked at compile-timeI Pattern matching to extract messagesI Transparently supports heterogeneous components

I Intra-machine: efficient message passing with copy-on-write semanticsI Inter-machine: TCP, UDP (soon), multicast (soon)I Special hardware components: GPUs via OpenCL

https://github.com/actor-framework19 / 27

https://github.com/actor-framework

Outline



3. Using VAST

4. Demo

19 / 27

Getting Up and Running

RequirementsI C++14 compiler

I Clang 3.4 (easiest bootstrapped with Robin’s install-clang)I GCC 4.9 (not yet fully supported)

I CMakeI Boost Libraries (headers only)I C++ Actor Framework (develop branch currently)

InstallationI git clone [email protected]:mavam/vast.git && cd vastI ./configure && make && make test && make installI vast -h # brief helpI vast -z # complete options

20 / 27

https://github.com/rsmmr/install-clang

https://github.com/mavam/vast

VAST Architecture

Export

IndexArchive

10.0.0.1 10.0.0.254 53/udp10.0.0.2 10.0.0.254 80/tcp

Import

21 / 27

DeploymentNetwork Transparency

I Actors can live in the same address space→ Efficiently pass messages as pointer

I Actors can live on different machines→ Transparent serialization of messages

Import with 2 Processes

Archive Index

Search

Receiver

Importer

One-Shot ImportImporter

Archive Index

Search

Receiver

22 / 27

Importing Logs

One-Shot ImportI vast -C -I -r conn.logI zcat *.log.gz | vast -C -II vast -C -I -p partition-2014-01 < conn.log

Import with 2 ProcessesI vast -C # coreI vast -I < conn.log # importer

23 / 27

VAST Architecture

Export

IndexArchive

10.0.0.1 10.0.0.254 53/udp10.0.0.2 10.0.0.254 80/tcp

Import

24 / 27

Synopsis: One-Shot Queries

JSON QueryI vast -C # coreI vast -E -o json -l 5 -q ':addr in 10.0.0.0/8'

Bro QueryI vast -C # coreI vast -E -o bro -l 5 -q ':addr in 10.0.0.0/8'

25 / 27

Outline



3. Using VAST

4. Demo

25 / 27

Thank You. . . Questions?

_ _____ __________| | / / _ | / __/_ __/| |/ / __ |_\ \ / /|___/_/ |_/___/ /_/


IRC at Freenode: #vast

27 / 27


VAST: Visibility Across Space and Time Architecture &...

Documents

Transcript of VAST: Visibility Across Space and Time Architecture &...