Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
-
Upload
sematext-group-inc -
Category
Data & Analytics
-
view
9.098 -
download
1
Transcript of Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
OCTOBER 13-16, 2016 • AUSTIN, TX
Large Scale Log Analytics with SolrRafał Kuć and Radu Gheorghe
Sematext Group
3
01About Us
RaduRafał
Logsene
4
02Agenda
Logstash + Solr
rsyslog + Solr
rsyslog + Redis + Logstash + Solr
Solr
5
01Flow in Logstash
/var/log/apache.log
redis
https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png
input
6
01Flow in Logstash
/var/log/apache.log
redis
https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png
plain
{json}
input
codec
7
01Flow in Logstash
/var/log/apache.log
redis
Rafał @kucrafal
grok{
"user": "Rafał","twitter": "@kucrafal"
}
- w $numberOfWorkers
https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png
plain
{json}
input
codec
filter
8
01Flow in Logstash
/var/log/apache.log
redis
Rafał @kucrafal
grok{
"user": "Rafał","twitter": "@kucrafal"
}
- w $numberOfWorkers
https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png
workers => 2
plain
{json}
input
codec
filter
output
9
01Simple Config https://github.com/sematext/lucene-revolution-samples/tree/master/2015
input {
file {
path => "/opt/logs/example.log"
start_position => "beginning"
}
}
output {
solr_http {
solr_url => "http://localhost:8983/solr/gettingstarted"
flush_size => 5000
workers => 4
}
}
bin/plugin install logstash-output-solr_http
apache combined logs
10
01Base Result
11
01Parse JSONinput {
file {
path => "/opt/logs/example.log.parsed"
start_position => "beginning"
…filter {
json {
source => "message"
}
}
output {
solr_http {
…
apache combined logs in JSON
bin/logstash -f logstash.conf -w 4 # filterWorkers=4
12
01JSON Result
input {
file {
path => "/opt/logs/example.log"
start_position => "beginning"
…filter {
grok {
match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
}
output {
solr_http {
…
13
01Grok
14
01Grok Result
15
01Flow Options
https://upload.wikimedia.org/wikipedia/commons/thumb/b/bb/Gorilla-server.svg/2000px-Gorilla-server.svg.pnghttps://www.elastic.co/assets/blt69f6410148efbab8/logstash.png
16
01Flow Options (cont.)
http://www.hanselman.com/blog/content/binary/Windows-Live-Writer/ef572a4c3e50_13F7B/redis_logo_a83f44f3-708d-4fad-aa6e-6eb0d6f82001.pnghttps://upload.wikimedia.org/wikipedia/commons/thumb/f/f8/Question_mark_alternate.svg/2000px-Question_mark_alternate.svg.png
or Kafka or *MQ or...
something light here
rsyslog
rsyslog
rsyslog
17
01Flow in rsyslog
/var/log/apache.log
syslog socket
input
18
01Flow in rsyslog
/var/log/apache.log
syslog socketmain queue (RAM+Disk)
inputqueue.typequeue.size...
19
01Flow in rsyslog
/var/log/apache.log
syslog socketmain queue (RAM+Disk)
inputqueue.typequeue.size...
queue.workerThreads(filter, parse and send events)
20
01Flow in rsyslog
/var/log/apache.log
syslog socketmain queue (RAM+Disk)
inputqueue.typequeue.size...
queue.workerThreads(filter, parse and send events)
queue.dequeueBatchSize
21
01Flow in rsyslog
/var/log/apache.log
syslog socketmain queue (RAM+Disk)
inputqueue.typequeue.size...
queue.workerThreads(filter, parse and send events)
queue.dequeueBatchSize
rsyslog_solr.py
rsyslog_solr.py
rsyslog_solr.py
action
template {JSON}
22
01Flow in rsyslog
/var/log/apache.log
syslog socketmain queue (RAM+Disk)
inputqueue.typequeue.size...
queue.workerThreads(filter, parse and send events)
queue.dequeueBatchSize
rsyslog_solr.py
rsyslog_solr.py
rsyslog_solr.py
action
template {JSON}
23
01Simple Config (1/2) https://github.com/sematext/lucene-revolution-samples/tree/master/2015
module(load="imfile")
module(load="omprog")
input(type="imfile"
File="/opt/logs/example.log"
Tag="apache:")
main_queue(
queue.highWatermark="100000"
queue.lowWatermark="50000"
queue.maxDiskSpace="5g"
queue.fileName="solr_action"
queue.spoolDirectory="/opt/rsyslog/queues"
queue.saveOnShutdown="on"
queue.workerThreads="4"
queue.dequeueBatchSize="500"
)
apache combined logs
24
01Simple Config (2/2)template(name="json_lines" type="list" option.json="on") {
constant(value="{")
constant(value="\"timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"message\":\"")
property(name="msg")
...
constant(value="\",\"syslog-tag\":\"")
property(name="syslogtag")
constant(value="\"}\n")
}
action(
type="omprog"
binary="/opt/rsyslog/rsyslog_solr.py"
template="json_lines"
)
get from https://github.com/rsyslog/rsyslog/tree/master/plugins/external/solr
25
01Base Result
26
01Base Result
15% rsyslog,4x1% rsyslog_solr.py
27
01Base Result
15% rsyslog,4x1% rsyslog_solr.py
125MB rsyslog, 4x15MB rsyslog_solr.pyDepends on queue. Here up to 100K events in RAM
28
01JSON Config# same main queue settings and modules
input(type="imfile"
File="/opt/logs/example.log.parsed"
Tag="apache:")
module(load="mmnormalize")
action(type="mmnormalize"
rulebase="/opt/rsyslog/json.rb"
)
template(name="json_lines" type="list") {
property(name="$!root") constant(value="\n")
}
action(type="omprog"
...
apache combined logsalready parsed in JSON
version=2
rule=:%root:json%
29
01JSON Result
30
01Normalizing Config
input(type="imfile"
File="/opt/logs/example.log"
Tag="apache")
action(type="mmnormalize"
rulebase="/opt/rsyslog/apache_combined.rb"
)
template(name="json_lines" type="list") {
property(name="$!all-json")
constant(value="\n")
}
version=2
rule=:%[
{"type": "word", "name": "clientip"},
{"type": "literal", "text": " "},
...
{"type": "char-to", "name": "agent", "extradata": "\""},
{"type": "literal", "text": "\""},
{"type": "rest", "name": "blob"}
]%
31
01Normalizing Result
32
01Normalizing “Should Scale”*
sys
tem log
d -ng
performance depends mostly on log length and not on the number of rules:http://blog.gerhards.net/2013/01/performance-of-liblognormrsyslog-parse.html
rule=apache_combined:%[
{"type": "word", "name": "clientip"},
...
{"type": "char-to", "name": "agent", "extradata": "\""},
{"type": "literal", "text": "\""},
{"type": "rest", "name": "blob"}
]%
rule=apache_common:%[
{"type": "word", "name": "clientip"},
...
{"type": "number", "name": "bytes"},
{"type": "rest", "name": "blob", "priority": 65535}
]%
...
33
01Normalizing with Five Rulesinput(type="imfile"
File="/opt/logs/example*"
Tag="apache")
action(type="mmnormalize"
rulebase="/opt/rsyslog/multiple_rules.rb"
)
if $!root <> "" then {
set $.final-json = $!root;
} else {
set $.final-json = $!all-json;
}
template(name="json_lines" type="list") {
property(name="$.final-json") constant(value="\n")
}
34
015 Rules Result
35
01OK, so this works:
rsyslog
rsyslog
rsyslog
36
01How about this:
rsyslog
rsyslog
rsyslog
37
01rsyslog.confmodule(load="imfile")
module(load="omhiredis")
input(type="imfile"
File="/opt/logs/example.log"
Tag="apache:")
template(name="json_lines" type="list" option.json="on") {...}
main_queue(queue.workerthreads="1"
queue.dequeueBatchSize="100"
queue.size="10000")
action(type="omhiredis"
mode="publish"
key="rsyslog_logstash"
template="json_lines")
./configure --enable-omhiredis
small&light queue
38
01logstash.conf
input {
redis {
data_type => "channel"
key => "rsyslog_logstash"
batch_count => 100
}
}
output {
solr_http {
...
}
}
JSON codec is implied
39
01Combined Result
rsyslog 1%
Redis 2%
Logstash 200%
rsyslog 10MB (10K queue)
Redis 1000MB (configurable)
Logstash 380MB
40
015-Rule Normalizing Result
rsyslog 100%
Redis 2%
Logstash 200%
rsyslog 30MB
Redis 1000MB
Logstash 450MB
41
01Shipper conclusions
rsyslog
rsyslog
rsyslog
rsyslog
rsyslog
rsyslog
easy setup; flexibleheavy
light; fastless flexible&easy
offloads buffers and Logstash processing;flexible and efficientsetup and maintenance overhead
42
01Solr Tuning Agenda
Schema and config adjustments
Time-based collections
Tiered cluster (e.g. hot vs cold nodes)
43
01Schema: Two Kinds of Fields
message:failed
"docValues": true"omitNorms": true,
"omitTermFreqAndPositions": true
44
01Schema: Two Kinds of Fields
message:failed
"docValues": true"omitNorms": true,
"omitTermFreqAndPositions": true
+20 to 100% capacity* 10% faster indexing*
* http://blog.sematext.com/2014/11/17/solr-presentations-lucene-solr-revolution/
45
01Commits
"updateHandler.autoSoftCommit.maxTime": 5000
"updateHandler.autoCommit.maxTime": 60000<ramBufferSizeMB>200</ramBufferSizeMB>
5s feels near-realtime while searching
Flush to disk every minute or 200MB
46
01Commits
"updateHandler.autoSoftCommit.maxTime": 5000
"updateHandler.autoCommit.maxTime": 60000<ramBufferSizeMB>200</ramBufferSizeMB>
5s feels near-realtime while searching
Flush to disk every minute of 200MB
+10% capacity; 10% faster indexing*
47
01Time-Based Collections
indexing, merges,most searches
doesn’t change => cache friendly can be optimized
delete without triggering merges
48
01Time-Based Collections
indexing, merges,most searches
doesn’t change => cache friendly=> can be optimized
delete without triggering merges
20-30x capacity; less indexing degradation*
* http://www.slideshare.net/sematext/side-by-side-with-elasticsearch-solr-part-2
49
01Tiered Cluster
hot1
hot2
cold1
cold2
cold3
cold4
50
01Tiered Cluster
hot1
hot2
cold1
cold2
cold3
cold4
51
01Tiered Cluster
hot1
hot2
cold1
cold2
cold3
cold4
ADDREPLICA
52
01Tiered Cluster
hot1
hot2
cold1
cold2
cold3
cold4
53
01Tiered Cluster
hot1
hot2
cold1
cold2
cold3
cold4
54
01Tiered Cluster
hot1
hot2
cold1
cold2
cold3
cold4
55
01Tiered Cluster
hot1
hot2
cold1
cold2
cold3
cold4
quick recent searches and indexing rare lengthy requests
56
01Tiered Cluster
cold1
cold2
cold3
cold4
quick recent searches and indexing rare lengthy requests
hot1
hot2
buffer for indexing spikes
57
01Tiered Cluster
cold1
cold2
cold3
cold4
quick recent searches and indexing rare lengthy requests
hot1
hot2
buffer for indexing spikes
less shards per collectionand the cluster is still balanced
58
01Tiered Cluster
cold1
cold2
cold3
cold4
quick recent searches and indexing rare lengthy requests
hot1
hot2
buffer for indexing spikes
less shards per collectionand the cluster is still balanced
CPU++
RAM++IO++
59
01Wrap-Up
60
01Wrap-Up
DocValues
commits
61
01Wrap-Up
DocValues
commits
https://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.pnghttps://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png
62
01Wrap-Up
DocValues
commits
https://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.pnghttps://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png
63
01Wrap-Up
DocValues
commits
http://www.funnyshirts.net/media/catalog/product/cache/1/image/9df78eab33525d08d6e5fb8d27136e95/z/o/zombies-hate-fast-food-funny-tshirt-preview.pnghttps://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.pnghttps://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png
64
01Wrap-Up
DocValues
commits
http://www.funnyshirts.net/media/catalog/product/cache/1/image/9df78eab33525d08d6e5fb8d27136e95/z/o/zombies-hate-fast-food-funny-tshirt-preview.pnghttps://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.pnghttps://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png
rsyslog
65
01Wrap-Up
DocValues
commits
http://www.funnyshirts.net/media/catalog/product/cache/1/image/9df78eab33525d08d6e5fb8d27136e95/z/o/zombies-hate-fast-food-funny-tshirt-preview.pnghttps://cdn0.iconfinder.com/data/icons/dance-fitness/72/13-512.pnghttps://www.standardlife.co.uk/resources/custom/uk/images/heroes/illustration/easy-box.png
rsyslog
rsyslog
rsyslog
rsyslog
66
01Questions?
Rafał Kuć@[email protected]
Radu [email protected]@sematext.com
Sematext@sematexthttp://sematext.com
67
01Questions?
Rafał Kuć@[email protected]
Radu [email protected]@sematext.com
Sematext@sematexthttp://sematext.com
we’re hiring, too!