Log analysis OpenSource con Logstash, Elasticsearch e Kibana
Using elasticsearch, logstash and kibana to create realtime ...
Transcript of Using elasticsearch, logstash and kibana to create realtime ...
![Page 1: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/1.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Alexander Reelsen @spinscale
Using elasticsearch, logstash and kibana to create realtime dashboards
![Page 2: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/2.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Agenda
• The need, complexity and pain of logging
• Logstash basics
• Usage examples
• Scalability
• Tools
• Demo
![Page 3: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/3.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
about
• Me Interested in metrics, ops and the web Likes the JVM Working with elasticsearch since 2011
• Elasticsearch, founded in 2012 Products: Elasticsearch, Logstash, Kibana, Marvel Professional services: Support & development subscriptions Trainings
![Page 4: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/4.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Why collect & centralise data?
• Access log files without system access
• Shell scripting: Too limited or slow
• Using unique ids for errors aggregate it across your stack
• Reporting (everyone can create his/her own report) Don’t be your boss’ grep/charting library
![Page 5: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/5.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Why collect & centralise data?
• Detect & correlate patterns Traffic, load, DDoS
• Scale out/down on-demand
• Bonus points: Unify your data to make it easily searchable
![Page 6: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/6.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Unify data
• apache
• unix timestamp
• log4j
• postfix.log
• ISO 8601
[23/Jan/2014:17:11:55 +0000]
1390994740
2009-01-01T12:00:00+01:00!2014-01-01
[2014-01-29 12:28:25,470]
Feb 3 20:37:35
![Page 7: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/7.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Enter logstash
• Managing events and logs
• Collect data
• Parse data
• Enrich data
• Store data (search and visualizing)
![Page 8: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/8.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Enter logstash
• Managing events and logs
• Collect data
• Parse data
• Enrich data
• Store data (search and visualizing)
} Input
} Output
} Filter
![Page 9: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/9.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Logstash architecture
Logstash
Input OutputFilter
? ?
![Page 10: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/10.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Inputs
collectd drupal_dblog elasticsearch eventlog exec file ganglia gelf gemfire generator graphite heroku imap irc jmx log4j lumberjack pipe puppet_facter rabbitmq redis relp s3 snmptrap sqlite sqs stdin stomp syslog tcp twitter udp unix varnishlog websocket wmi xmpp zenoss zeromq
![Page 11: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/11.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Outputs
boundary circonus cloudwatch csv datadog elasticsearch exec email file ganglia gelf gemfire google_bigquery google_cloud_storage graphite graphtastic hipchat http irc jira juggernaut librato loggly lumberjack metriccatcher mongodb nagios null opentsdb pagerduty pipe rabbitmq redis riak riemann s3 sns solr_http sqs statsd stdout stomp syslog tcp udp websocket xmpp zabbix zeromq
![Page 12: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/12.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Installation
• ruby application, but Java required (JRuby)
• Download tarball, deb, RPM (also repositories) no gem/dependency hell!
• Puppet module
![Page 13: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/13.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Simple setup
• Download, create config and run
input {! stdin {}!}!!output {! stdout { codec => rubydebug }!}
echo foo | logstash-1.4.0.rc1/bin/logstash -f simple.conf!{! "message" => "foo" ! "@version" => "1" ! "@timestamp" => "2014-01-20T13:30:59.648Z" ! "host" => "kryptic.fritz.box"!}
simple.conf
![Page 14: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/14.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Analyze the output{! "message" => "foo" ! "@version" => "1" ! "@timestamp" => "2014-01-20T13:30:59.648Z" ! "host" => "kryptic.fritz.box"!}
• message: Original content
• version: internal
• timestamp: Current timestamp
• host: Logstash hostname
![Page 15: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/15.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
But what about filtering?
input {! stdin {}!}!!filter {! grok {! match => [ "message" "%{WORD:firstname} %{WORD:lastname} %{NUMBER:age}" ]! }!}!!output {! stdout { codec => rubydebug }!}
![Page 16: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/16.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
But what about filtering?
echo "Alexander Reelsen 30" | logstash-1.4.0.rc1/bin/logstash -f sample-2.conf!{! "message" => "Alexander Reelsen 30" ! "@version" => "1" ! "@timestamp" => "2014-01-21T16:56:02.502Z" ! "host" => "kryptic" ! "firstname" => "Alexander" ! "lastname" => "Reelsen" ! "age" => "30"!}
![Page 17: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/17.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Grok
• Maintaining regexes for mere mortals http://logstash.net/docs/1.3.3/filters/grok
• Default patterns ciscofw, haproxy, apache, syslog, cron, nagios, postfix, redis... !https://github.com/logstash/logstash/tree/v1.3.3/patterns
• Grok Debugger https://grokdebug.herokuapp.com/
![Page 18: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/18.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Syslog example with grok
input { stdin {} }!!filter {! grok {! match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }! }! date {! match => [ "syslog_timestamp", ! "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]! }!}!!output { stdout { codec => rubydebug } }
![Page 19: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/19.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Syslog example with grok
cat sample-syslog.txt| logstash-1.4.0.rc1/bin/logstash -f sample-syslog.conf!{! "message" => "Jun 10 04:04:01 lvps109-104-93-171 postfix/smtpd[11105]: connect from mail-we0-f196.google.com[74.125.82.196]" ! "@version" => "1" ! "@timestamp" => "2014-06-10T04:04:01.000+02:00" ! "host" => "kryptic.local" ! "syslog_timestamp" => "Jun 10 04:04:01" ! "syslog_hostname" => "lvps109-104-93-171" ! "syslog_program" => "postfix/smtpd" ! "syslog_pid" => "11105" ! "syslog_message" => "connect from mail-we0-f196.google.com[74.125.82.196]"!}
![Page 20: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/20.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Syslog example with grok
cat sample-syslog.txt| java -jar logstash-1.3.3-flatjar.jar agent -f sample-syslog.conf!{! "message" => "Jun 10 04:04:01 lvps109-104-93-171 postfix/smtpd[11105]: connect from mail-we0-f196.google.com[74.125.82.196]" ! "@version" => "1" ! "@timestamp" => "2014-06-10T04:04:01.000+02:00" ! "host" => "kryptic.local" ! "syslog_timestamp" => "Jun 10 04:04:01" ! "syslog_hostname" => "lvps109-104-93-171" ! "syslog_program" => "postfix/smtpd" ! "syslog_pid" => "11105" ! "syslog_message" => "connect from mail-we0-f196.google.com[74.125.82.196]"!}
Jun 10 04:04:01 lvps109-104-93-171 postfix/smtpd[11105]: connect from mail-we0-f196.google.com[74.125.82.196]
![Page 21: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/21.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Filters
advisor alter anonymize checksum cidr cipher clone collate csv date dns drop elapsed elasticsearch environment extractnumbers fingerprint gelfify geoip grep grok grokdiscovery i18n json json_encode kv metaevent metrics multiline mutate noop prune punct railsparallelrequest range ruby sleep split sumnumbers syslog_pri throttle translate unique urldecode useragent uuid wms wmts xml zeromq
![Page 22: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/22.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Codecs
cloudtrail compress_spooler dots edn edn_lines fluent graphite json json_lines json_spooler line msgpack multiline netflow noop oldlogstashjson plain rubydebug spool
![Page 23: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/23.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
JSON codecinput {! stdin {! codec => json! }!}!!output {! stdout { codec => rubydebug }!}
(echo -e '{"foo":"bar", "spam" : "eggs"\n} ' ) | logstash-1.4.0.rc1/bin/logstash -f sample-json-codec.conf!{! "foo" => "bar" ! "spam" => "eggs" ! "@version" => "1" ! "@timestamp" => "2014-01-23T13:12:17.325Z" ! "host" => "kryptic.local"!}
![Page 24: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/24.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
JSON lines codecinput { stdin { codec => json_lines } }!output { stdout { debug => true } }
(echo -e '{"foo":"bar", "spam" : "eggs" }' ; echo '{ "c":"d", "e": "f" }') | logstash-1.4.0.rc1/bin/logstash -f sample-json-multi-codec.conf!{! "foo" => "bar" ! "spam" => "eggs" ! "@version" => "1" ! "@timestamp" => "2014-01-23T13:17:47.582Z" ! "host" => "kryptic.local"!}!{! "c" => "d" ! "e" => "f" ! "@version" => "1" ! "@timestamp" => "2014-01-23T13:17:47.584Z" ! "host" => "kryptic.local"!}
![Page 25: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/25.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
CLF log files
input { stdin {} }!!filter {! grok {! match => [ message "%{COMBINEDAPACHELOG}" ]! }!}!!output { stdout { codec => rubydebug } }
193.99.144.85 - - [23/Jan/2014:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"!!193.99.144.85 - - [23/Jan/2014:17:11:55 +0000] "GET /myimage.jpg HTTP/1.1" 200 140 "-" "Googlebot"
![Page 26: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/26.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
CLF log files{! "message" => "193.99.144.85 - - [23/Jan/2014:17:11:55 +0000] \"GET / HTTP/1.1\" 200 140 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19\"" ! "@version" => "1" ! "@timestamp" => "2014-01-24T07:56:02.460Z" ! "host" => "kryptic.local" ! "clientip" => "193.99.144.85" ! "ident" => "-" ! "auth" => "-" ! "timestamp" => "23/Jan/2014:17:11:55 +0000" ! "verb" => "GET" ! "request" => "/" ! "httpversion" => "1.1" ! "response" => "200" ! "bytes" => "140" ! "referrer" => "\"-\"" ! "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19\""!}
![Page 27: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/27.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Write to elasticsearch
input { stdin {} }!!filter {! grok {! match => [ message "%{COMBINEDAPACHELOG}" ]! }!}!!output {! elasticsearch {! protocol => 'http'! }!}
![Page 28: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/28.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Use case: Log files
Shipper Logstash Store/Search Visualize
![Page 29: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/29.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Use case: Log files with broker
Shipper Logstash Store/Search
Visualize
Broker
![Page 30: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/30.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Use case: Log files with broker
Shipper Logstash Store/Search
Visualize
Broker
Shipper
Shipper
![Page 31: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/31.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Scale out any component
Shipper Logstash Store/Search
Visualize
Broker
Shipper
Shipper
Broker
Broker
![Page 32: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/32.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Scale out any component
Shipper Logstash Store/Search
Visualize
Broker
Shipper
Shipper
Broker
Broker
Logstash
Logstash
![Page 33: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/33.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Scale any component
Shipper Logstash Store/Search
Visualize
Broker
Shipper
Shipper
Broker
Broker
Logstash
Logstash
Store/Search
![Page 34: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/34.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Logstash scaling
• Events get passed via ruby SizedQueue
• input/worker/output threads, can be configured
• each input is one thread, unless explicitly configurable
• one worker thread by default, use -w to change
• output is a single thread (some outputs have their own queueing thread)
!http://logstash.net/docs/1.3.3/life-of-an-event
![Page 35: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/35.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
data
time
Data growth & capacity planning
![Page 36: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/36.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
data
time
Data growth & capacity planning
No!
![Page 37: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/37.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Data growthda
ta
time
![Page 38: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/38.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Data growth & capacity planningda
ta
time
?
![Page 39: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/39.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Data growth & capacity planning
• Added a new forwarder/shipper
• Added new type of logs
• Increased traffic/usage
!
• Capacity planning? data
time
![Page 40: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/40.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Capacity managementda
ta
time
capacity of one node
![Page 41: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/41.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Scale data to your needs!
per monthlogs-2014-01
1
• Small dataset
• Fits on one machine, cannot be divided
![Page 42: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/42.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Scale data to your needs!logs-2014-02-w01
1 2
logs-2014-02-w04
1 2 per week...
• More data gets indexed
• Can be scaled on up to eight machines
![Page 43: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/43.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Scale data to your needs!logs-2014-03-01
1 1
logs-2014-03-31
1 1 per day...
• Safety: Data available twice in cluster
• Can be scaled on up to 62 machines
![Page 44: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/44.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Scale data to your needs!
logs-2014-02-w01
1 2
logs-2014-02-w04
1 2
logs-2014-03-01
1 1
logs-2014-03-31
1 1
per month
per week
per day
...
...
logs-2014-01
1
![Page 45: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/45.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana
![Page 46: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/46.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana
![Page 47: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/47.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana
![Page 48: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/48.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana
![Page 49: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/49.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana
![Page 50: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/50.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Tools
![Page 51: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/51.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Useful helpers
• Curator http://www.elasticsearch.org/blog/curator-tending-your-time-series-indices/
• Puppet module https://github.com/elasticsearch/puppet-logstash
• logstash forwarder https://github.com/elasticsearch/logstash-forwarder
• Logstash cookbook http://cookbook.logstash.net/
![Page 52: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/52.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Demo - Meetup RSVP stream
![Page 53: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/53.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Soon... 1.4
• tons of documentation updates
• puppet module love
• tests to ensure backwards compatibility
• new packaging (less startup time)
![Page 54: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/54.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Thanks for listening
![Page 55: Using elasticsearch, logstash and kibana to create realtime ...](https://reader033.fdocuments.in/reader033/viewer/2022042503/586cd08d1a28abb26b8bee92/html5/thumbnails/55.jpg)
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Q & A
Alexander Reelsen @spinscale
P.S. We’re hiring http://elasticsearch.com/about/jobs http://elasticsearch.com/support