Logging logs with Logstash

Steve Howe – RetailMeNot UK Ltd.

showe@rmn.com

About me..

Devops engineer at RetailMeNot Obsessed with making things betterHusband, father, frequently busy

Why Logstash?

Jordan Sissel

Why Logstash?

Jordan SisselNoticed a problem

Why Logstash?

Jordan SisselNoticed a problem

Why Logstash?

Jordan SisselNoticed a problem

Why Logstash?

Jordan SisselNoticed a problem

Why Logstash?

Jordan SisselNoticed a problem

What does Logstash provide?

Empowerment

What does Logstash provide?

EmpowermentAggregation

What does Logstash provide?

EmpowermentAggregationSearch

What does Logstash provide?

EmpowermentAggregationSearch

What is Logstash (ELK stack)?

Centralization

What is Logstash (ELK stack)?

CentralizationVisualization

What is Logstash (ELK stack)?

CentralizationVisualizationSegregation

Components

httpd Logstashshipper

redis Logstashindexer

Elasticsearch

Kibana

MySQL Logstashshipper

Logstash-forwarder shipper

Small package (java) (used to be “lumberjack”)

Logstash-forwarder shipper

Small package (java) (used to be “lumberjack”)Grok processing can happen on client

Logstash-forwarder shipper

Small package (java) (used to be “lumberjack”)Grok processing can happen on clientOne shipper, multiple logfiles (perms)

Logstash-forwarder shipper

Small package (java) (used to be “lumberjack”)Grok processing can happen on clientOne shipper, multiple logfiles (perms)Many log formats, one output - JSON

Redis

Message broker

Redis

Message brokerSSL encryption for non-VPN networks

Redis

Message brokerSSL encryption for non-VPN networksScalable

Logstash Indexer

Java app

Logstash Indexer

Java appGrok processing of logs off app-servers

Logstash Indexer

Java appGrok processing of logs off app-serversOutputs to multiple endpoints, if required

Logstash Indexer

Java appGrok processing of logs off app-serversOutputs to multiple endpoints, if requiredScalable

Elasticsearch

Mature, scalable db

Elasticsearch

Mature, scalable dbDocument-based index, Query DSL API

Elasticsearch

Mature, scalable dbDocument-based index, Query DSL APISharded, clustered

Elasticsearch

Mature, scalable dbDocument-based index, Query DSL APISharded, clusteredIndices are stored in date format

Kibana

Standalone (used to be simple website)

Kibana

Standalone (used to be simple website)Points directly at the Elasticsearch db

Kibana

Standalone (used to be simple website)Points directly at the Elasticsearch dbSolr Lucene syntax

Kibana

Standalone (used to be simple website)Points directly at the Elasticsearch dbSolr Lucene syntaxScalable

Kibana

Standalone (used to be simple website)Points directly at the Elasticsearch dbSolr Lucene syntaxScalable

Demo

Tricks

Elasticsearch Curator, single indices

Tricks

Elasticsearch Curator, single indicesOutputs to Graphite, Nagios, Cloudwatch,

Datadog, GELF, Slack, Hipchat, PagerDuty etc.

Tricks

Elasticsearch Curator, single indicesOutputs to Graphite, Nagios, Cloudwatch,

Datadog, GELF, Slack, Hipchat, PagerDuty etc.Kibana dashboards

Gotchas

Logstash Elasticsearch client (versions, AWS)

Gotchas

Logstash Elasticsearch client (versions, AWS)Enterprise ready? (utter lack of users/security,

streams, alerting)

Gotchas

Logstash Elasticsearch client (versions, AWS)Enterprise ready? (utter lack of users/security,

streams, alerting)Graylog – Alternative log engine

Links

Jordan Sissels Github - https://github.com/jordansisselJordan Sissel pres - http://semicomplete.com/presentations/logstash-scale11x/#/33 Logstash book - http://www.logstashbook.com/ Enterprise ready? (users, streams, alerting)

Cheers!