Log Management With Open Source Tools

8/10/2019 Log Management With Open Source Tools

1/21

Log Management with Open-Source Tools

Risto Vaarandirvaarandi 4T Y4H00 0T !0M


2/21


3/21

"h# collect logs %rom #our &Ts#stem and networ'$

O(servation ) logs contain in%ormation which is o%tennot availa(le %rom other sources

Real-time monitoring ) anal#*e logs in real-time +ornear-real-time, %ashion in order to discover importantchanges in the state o% the &T s#stem

.ost-%actum incident anal#sis ) leverage collecteddata %or discovering un'nown past incidents andgetting detailed insights into them


4/21

"h# use open source tools %or logmanagement$

!ommercial S&/M and log management %ramewor's

man# %ramewor's are consultant-oriented ) have comple1 design andinsu%%icient documentation

prohi(itive deplo#ment and licensing costs

man# %ramewor's repeat a num(er o% design mista'es o% networ'management solutions +made almost two decades ago2,

.ast e1perience with networ' management solutions

.hase 3 initial mar'eting h#pe %ollowed (# a num(er o% success storiesin the conte1t o% large and wealth# institutions

.hase disappointment among man# potential customers +%aileddeplo#ments prohi(itive pricing etc5, and search %or alternatives

.hase6 appearance o% well-designed open-source solutions which(ecome widel# used and ac'nowledged especiall# (# small- and mid-si*e enterprises


5/21

Traditional log collection protocols

The scene o% log collection protocols was relativel#sta(le %or two decades

7S s#slog ) the onl# cross-vendor protocol designedspeci%icall# %or logging

8. (ased plainte1t thus resource-e%%icient (utunrelia(le and not secure

Simple message la#out in the 8. %rame ) priorit#

simple timestamp host name program nameunstructured message te1t


6/21

9ew log collection protocols

&/T: s#slog +00;, ) support %or including structureddata in messages 8. and T!. (ased transportencr#ption and authentication detailed timestamps

!// +!ommon /vent /1pression, logging standard+03, ) use /L: S9M.trap messages etc5


7/21

/1amples

? Traditional 7S s#slog ) priorit# value @ encapsulates %acilit# value 6? +daemon, and severit# value 4 +warning, 6A@ B 4 C @

D@E9ov 3F 366G;m#hostids3;;I port scan from 192.168.1.102

? &/T: s#slog ) note high granularit# timestamps with time*one in%ormation

? and two (loc's o% structured data

D@E3 03-33-3FT366G;56B000 m#hostids3;;-timeJualit# t*KnownC3 isS#ncedC3Iorigin ipC3053535Iport scan from 192.168.1.102

? !// message %ormat ) use standard 7S s#slog message %or transporting? structured data in


8/21

"h# pass structured data in logmessages$

8nstructured message %ields o%ten contain additionalin%ormation a(out event which needs to (e highlighted

&t is much easier to parse structured data +'e#word-value pairs, than unstructured %ree-%ormat strings

Some structured data can (e used without e1traparsing )


9/21

9

Log collection on Linu1 plat%orm

local

programs

s#slog

server

networ'port

=dev=log

incoming messages

messages to

remote s#slogservers

messages %rom other nodes

=etc=s#slog-server5con%

con%iguration

=proc='msg

'ernel

=var=log=555

to local log%ilesopenlog+6,s#slog+6,555

d( >8&


10/21

S#slog servers ) rs#slog

http==www5rs#slog5com

B %ast message processing e%%icient multithreading designed to handleat least 3G0-00K messages per second +see the paper QRs#slog goingup %rom 40K messages per second to G0K (# Rainer >erhards %romLinu1 Kongress 030,

B (ac'wards compati(le with 89& s#slogd con%iguration directives

B has a num(er o% uniue %eatures and advantages over competitors+dis' (ased (u%%ers support %or /lasticsearch data(ase etc5,

- documentation could (e (etter

- con%iguration language has a non-intuitive s#nta1

- %iltering conditions can not (e named which prevents their reuse


11/21

S#slog servers ) s#slog-ng

http==www5(ala(it5com=networ'-securit#=s#slog-ng=

B a %le1i(le and reada(le con%iguration language which allows %orspeci%#ing comple1 con%igurations

B single-threaded until the 65 version (ut multi-threading has (een

introduced into recent versions which considera(l# improves scala(ilit#and per%ormance

B well documented

- open-source edition does not support dis' (ased (u%%ers

- no support %or /lasticsearch +although could (e con%igured through a

sel%-developed output plugin,


12/21

S#slog servers ) n1log

http==n1log-ce5source%orge5net=

B native support %or "indows plat%orm and "indows /vent Log

B supports the use o% em(edded .erl constructs %or message processing

B supports a num(er o% input and output t#pes not supported (#

competitors +e5g5 accepting input events %rom SJL data(asesproducing output events in >/L: %ormat etc5,

- poor message %iltering per%ormance


13/21

/lasticsearch 7 %or logmanagement

http==www5elasticsearch5org=

Upache Lucene (ased noSJL data(ase technolog# that is %reuentl#used %or storing log data

native support %or distri(uted operations and (uilding clusters

allows %or splitting inde1es into parts +shards, and distri(uting shardsover several nodes +e5g5 split an inde1 into shards and distri(ute themover nodes turning dis's at individual nodes into a single logicalstorage space,

inde1es can (e con%igured to have one or more replicas which increases%ault tolerance +e5g5 split an inde1 into shards and con%igure the inde1to have 3 replica and distri(ute resulting 4 shards across 4 nodes,

(uiltin support %or data compression +important when storing largevolumes o% log data,

supported (# several log management tools +Ki(ana >ra#loglogstash rs#slog,


14/21

Log management tools ) Ki(ana

http=='i(ana5org=

Ki(ana is a >8& %or searching log data stored into /lasticsearch 7

Ki(ana is designed to wor' with logstash log preprocessing tool (ut canaccept data %rom an# other tool which is a(le to store it to /lasticsearch

in a recogni*a(le wa# +e5g5 rs#slog, Ki(ana is lightweight written in Ru(# accessi(le over HTT. and

contains onl# searching and reporting %unctionalit# +e5g5 userauthentication and SSL connectivit# has to (e accomplished withe1ternal tools li'e Upache reverse pro1#,

"hen (uilding a Ki(ana (ased log management solution #ou arecreating the s#stem %rom well-documented and well-esta(lished (uilding(loc's and thus having the opportunit# %or man# customi*ations duringinitial installation and later maintenance


15/21

Ki(ana we( inter%ace


16/21

Log management tools ) >ra#log

http==gra#log5org=

U %ull log management solution consisting o% a server %or log messagereception +s#slog >/L:, and a >8&

The >8& is user-%riendl# with (uiltin help and is intuitive to use

Man# con%iguration tas's +such as setting log data retention intervalsetc5, can (e accomplished through a we( inter%ace

>ra#log supports users with di%%erent roles and passwordauthentication

/arlier versions o% >ra#log emplo#ed single-server approach which

limited the s#stem scala(ilit# while most recent versions allow to runseveral servers in parallel


17/21

>ra#log we( inter%ace


18/21

Other log management tools

Logstash +http==www5logstash5net=, - has a we( inter%ace %orsearching logs stored to /lasticsearch data(ase (ut since itsupports large num(er o% input and output t#pes it is mostl#used as a log parsing and conversion tool

/LSU +http==code5google5com=p=enterprise-log-search-and-archive=, - a log management s#stem which is (uilt on top o%s#slog-ng M#SJL and Sphin1


19/21

9et%low protocol

.roposed (# !isco in 3;;0s nowada#s supported (#man# maor vendors

U 9et%low-ena(led networ' device +e5g5 router switchdedicated pro(e, collects networ' tra%%ic statistics ande1ports it to collector over 8.

Tra%%ic statistics consists o% %low records where eachrecord descri(es some network flow

9etwor' %low ) unidirectional seuence o% pac'etswhich share transport protocol source and destination&. source and destination port and %ew otherparameters +e5g5 t#pe o% service,


20/21

/1ample o% collected 9et%low data

The %ollowing two records represent a success%ull# negotiatedand completed T!. connection %rom client 30565353 port 4@@;to the HTT. service +port @0, running at the server 3055353

StartC 036-0-3@ 00040G5F66DurationC 05034

T!. 305653534@@; -E 3055353@0

TCPflagsC 5U.5S:PacketsC GBytesC G36

StartC 036-0-3@ 00040G5F64DurationC 05030

T!. 3055353@0 -E 305653534@@;

TCPflagsC 5U.5S:PacketsC 4BytesC 6FG


21/21

How to collect=use net%low data

/na(le 9et%low collection at #our networ' device oruse dedicated pro(es +e5g5 %pro(e,

Open-source so%tware pac'ages %or collecting 9et%low

9%Sen +http==n%sen5source%orge5net=, SiLK +http==tools5netsa5cert5org=sil'=,

:low-tools +http==www5splintered5net=sw=%low-tools=, - unmaintained

"hat #ou might (e interested in %inding in 9et%low data

:lows with unusual com(inations o% T!. %lags +e5g5 :&9 without U!K, :lows which represent connections to=%rom 'nown (ad &. addresses

8ne1pected spi'es in tra%%ic volumes +measured in num(er o% (#tespac'ets %lows, associated with certain sources +e5g5 %oreign &.addresses or (ad &. addresses,

Log Management With Open Source Tools

Documents

Transcript of Log Management With Open Source Tools