On Centralizing Logs
39
Click here to load reader
-
Upload
sematext-group-inc -
Category
Technology
-
view
120 -
download
2
description
On Centralizing Logs with Syslog, LogStash, Elasticsearch, Kibana. Presentation from Radu Gheorghe from Sematext at Monitorama EU 2013.
Transcript of On Centralizing Logs
Hello World!
Logsene
mlmoneu13cf for -44%
app
app
app
app
files
files
app
app
app
app
files
filesElasticsearchlogstash
Kibana
Elasticsearch Reason #1: Quick Search
No indexing
But...
=>
...and other reasons
good write speed lots of tools for logging
scales easily
Production Tips
stability performance
Stability 1/4: Discovery
multicast unicast
vs
cluster name list of nodes
+ plugins: EC2, GCE
Stability 2/4: Preventing Split Brain
minimum_master_nodes = N/2 + 1
Stability 3/4: No OOMs, pls!
1GB
½ total RAM
Monitor the requirements
SPM for Elasticsearch
20% off with MONEU2013
Stability 4/4: Field Cache
can be changed to
index.cache.field.type: soft
indices.fielddata.cache.size: X%
Performance 1/4: Bulk Processing
use Bulk APIor Bulk UDP API
...translog.flush_threshold_ops
Performance 2/4: Refresh Interval
http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/
default:every second => but
every 5s+25% indexing*
every 30s+70% indexing*
Performance 3/4: Timed Indices
Performance 4/4: Buffers
...index_buffer_size: 30%(YMMV)
index.store.type: mmapfs(on 64-bit machines)
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Setting Up Kibana as Frontend
servers you
Kibana: Search
Kibana: Visualize
Meet Some Syslog Daemons
syslogdtraditional
everywhere
syslog-ngOSE, PE
documentation++config format++
rsyslogOSS onlyES output*
* http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
X-ray of a Modern Syslog Daemon
read+bufferfile
/dev/log…
parsesyslog formats
JSONunstructured data
assembleconditionalsformatting
...
buffer+writefile
syslogElasticsearch
...
2001's RFC3164: The Semi-Standard
<10>Oct 11 22:14:15 host program:hello world
TCP + LF =
no year, ms, nor TZ
little structure
2009's RFC5424
<165>1 2003-10-11T22:14:15.003Z host program - - - [origin ip="192.168.0.1"] hello world
[ structured=data ] octet-count* + LF =
* UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)
Teaching Old Dog New Tricks
RSYSLOG_ForwardFormat(ISO8601 over RFC3164)
$MaxMessageSize 2048klog_message_size(2097152)
@cee: {"message": "hello world"} @@(o)192.168.0.1
octet-counted framing
Reliable Transport? Encryption?
TCP + TLS (RFC5425)
RLTP + TLS RELP + TLS
Logstash: The Swiss Army Knife
inputs(+codecs)
filters(parse, modify)
outputs(+codecs)
lots of plugins => lots of options
Logstash: Example
Lumberjack
Logstash Elasticsearch
Logstash: Add Buffer
Lumberjack
Lumberjack
Logstash: Scale Everything
Lumberjack
Lumberjack
Lumberjack
Lumberjack
Back to the Beginning
Lumberjack
Lumberjack
Lumberjack
Lumberjack
syslogd
Logsene
Lumberjack
Lumberjack
Lumberjack
Lumberjack
syslogd
Logsene
http://sematext.com/logsene
(More) Alternatives
files
syslog
Alternatives Can Mix
files
syslog
LogstashElasticsearch Kibana
rsyslog 1/4: Upgrade to 7.x
RPMs or DEBs better performance
nicer config format omelasticsearch
rsyslog 2/4: Faster Inputs
UDPincrease TimeRequery
TCPuse imptcp
rsyslog 3/4: Main Message Queue
$MainMsgQueueType FixedArray$MainMsgQueueSize 1000000....
...or LinkedList or Disk
$...DequeueBatchSize 1000 $...WorkerThreads 3
rsyslog 4/4: Action Queue
queue.type="linkedlist" queue.size="1000000"
bulkmode="on" # ES specific queue.dequeuebatchsize="1000"
queue.workerthreads="3"
