Log Management with ELKELASTICSEARCH, LOGSTASH, KIBANA FOR CENTRALIZED LOGS

Purpose Centralized Log Management

◦ Collect, Parse and Filter using Logstash◦ Store, Index and Search using Elasticsearch◦ Visualize using Kibana

Full open source stack◦ Use for free◦ Support plan from Elasticsearch company

Elasticsearch Real-time search engine

◦ Based on Apache Solr/Lucene◦ Pure Java◦ Document database

◦ Advanced text indexing◦ Fuzzy search

◦ Replication/Sharding for true scalability

Logstash JRuby Based log processor Pluggable event pipeline

◦ Input plugins◦ Filter plugins◦ Codec plugins◦ Output plugins

DevOps Comunity◦ Mix of developers, operations and system administrators

Kibana Browser based dashboard for ElasticSearch Visualization of query results

◦ Time Charts◦ Filter any field◦ Compare subsets

Logstash pipeline Define input, filters and outputs

Simple configuration file

Ruby syntax

Logstash not just for logs Interpretes different log formats

◦ Syslog messages◦ Log4j with full details◦ Apache log files

Other event types too◦ Ganglia server monitoring events◦ SNMP events◦ Windows EventLog

Pre-proces before sending◦ lumberjack

Logstash Inputs Rsyslog via TCP/UDP Log4j appender JMX Listener Logstash-forwarder File tails SNMP Ganglia …

Logstash Filters

Grok Filter Readable regex Predefined patterns for common log data Extract to properties

◦ Indexed properties

Metrics Filter Aggregate metrics

◦ Event rate using sliding windows◦ 1 min◦ 5 min◦ 15 min

◦ Min/max/stddev/percentiles

Logstash Outputs

Statsd output Node.js based

◦ Counters ◦ Timers◦ Graphite frontend

Alert outputsSend alerts

◦ Email◦ Pagerduty◦ XMPP/Jabber◦ Hipchat◦ Nagios

Use treshold from metrics filter

ElasticSearch output Auto-creates new index per day

Index all recognized fields

Full text index, customizable indexer, mapper

Kibana dashboard

Scalability Easy deployment using chef/puppet/docker