LOGSTHE MISSING GOLD MINE

by Franjo Žilić

WHY TALK ABOUT LOGS?They are all around us

We use them to debug our software every day

Is that all tere is?

CONVENTIONAL WAY

CONVENTIONAL WAY

CONVENTIONAL WAY grep -v " 200 " access_log-20160124

HOW ABOUT A DIFFERENT VIEW

ELK

* not that elk.

ELKMeet the familiy

ELKelasticsearchlogstash*

kibana* or something else

ELASTICSEARCHThe database

document orienteddistributedshardedreplicated*

timestamp partitioning*

Java & lucene

KIBANAThe UI

colorfulsmallGRAPHS

LOGSTASHThe parser

uses GROK

plugins for nearly everything

PARSING LOGSfilter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } }

PARSING LOGS(̂?#regex designed to parse VyOS kernel log)(?#some global parsing, like timestamp, fitlter, interfaces, and so on)(?<time>[̂ ]* [̂ ]* [ ̂]*) (?<host>[̂ ]*) (?<vyos_sylog_facility>[̂: ]*)?: \[(?<vyos_fw_filter_name>[̂\[]*)\] ?IN=( |(?<vyos_in_interface>[̂ ]*) )OUT=( |(?<vyos_out_interface>[̂ ]*) )(MAC=( |(?<vyos_mac_addres>[̂ ]*) ))?SRC=( |(?<vyos_source_ip_address>[̂ ]*) )DST=( |(?<vyos_dstination_ip_address>[̂ ]*) )LEN=( |(?<vyos_len>[̂ ]*) )TOS=( |(?<vyos_tos>[̂ ]*) )PREC=( |(?<vyos_prec>[̂ ]*) )TTL=( |(?<vyos_ttl>[̂ ]*) )ID=( |(?<vyps_packet_id>[̂ ]*) )(?<vyos_packet_flags>[̂ |(PROTO)]*)? ?PROTO=( |(?<vyps_ip_protocol>[̂ ]*))(?#here comes the fun part, different parser for different interesting packet types, regex if and positive lookbehind matching each type of interesting )(?:(?<=(TCP))((?#tcp specific matchers) ?SPT=( |(?<vyos_source_port>[̂ ]*) )DPT=( |(?<vyos_destination_port>[̂ ]*) )WINDOW=( |(?<vyos_tcp_window>[̂ ]*) )RES=( |(?<vyos_res>[̂ ]*) )(?<vyps_tcp_state>[̂(URGP)]* ).*)|(?:(?<=(UDP))((?#udp specific matchers) ?SPT=( |(?<vyos_source_port>[̂ ]*) )DPT=( |(?<vyos_destination_port>[̂ ]*) ).*)|(?:(?<=(ICMP))((?#icmp specific matchers) TYPE=( |(?<vyos_icmp_type>[̂ ]*) )CODE=( |(?<vyos_icmp_code>[̂ ] )).*)|(.*))))$

CUSTOM SOLUTION?Implement custom data collection within the application

Populate data with Servlet filters or Spring AOP

Index data in Elasticsearch

BENEFITSLogging that meets your needs

Ability to extract analytical data

Near real-time event tracking

REAL WORLD EXAMPLESTracking HTTP traffic

Tracking user activity

Tracking 3rd party API responses

HTTP DATABandwith costs money

Web site scrapers are common

Serving non-compressed data is expensive

Identification of scrapers can reduce cost

3RD PARTY API TRACKINGLog all requests and responses

Monitor performance

Monitor availability

Provide extra troubleshooting data

REAL WORLD DATA

BENEFITS OVER COMMERTIALSOLUTIONSno software coston site or cloud installfull control over datafully customizable

DRAWBACKSadditional setupnothing is predefinedno "enterprise" support

CONCLUSIONSUtilize data that you already have

learn more abiout your users and applications

THANK YOUelastic.co

Fluentd

Apache Flume