Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann
-
Upload
danny-abukalam -
Category
Software
-
view
460 -
download
4
Transcript of Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann
![Page 1: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/1.jpg)
![Page 2: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/2.jpg)
Unravelling LogsMatt Jarvis - Head of Cloud Computing @ DataCentred
![Page 3: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/3.jpg)
Traditional log file analysis ...
● Troubleshooting● Post incident forensics● Security auditing● Reporting and analysis
![Page 4: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/4.jpg)
![Page 5: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/5.jpg)
![Page 6: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/6.jpg)
![Page 7: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/7.jpg)
![Page 8: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/8.jpg)
Nova Controller :
● nova-api.log● nova-cert.log● nova-conductor.log● nova-scheduler.log
Glance Server :
● api.log● image-cache.log● registry.log
Neutron Controller :
● openvswitch-agent.log● server.log
Network Node :
● openvswitch-agent.log● neutron-ns-metadata-proxy*.log● metadata-agent.log● dhcp-agent.log
Compute Node :
● openvswitch-agent.log● nova-compute.log
![Page 9: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/9.jpg)
● INGEST CENTRALLY
● STRUCTURE
● INDEX
● ANALYZE
![Page 10: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/10.jpg)
![Page 11: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/11.jpg)
![Page 12: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/12.jpg)
● Distributed search engine● Highly scalable● Super fast● HTTP interface
![Page 13: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/13.jpg)
FIXME Kibana screenshot
![Page 14: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/14.jpg)
● Collect● Parse● Transform
![Page 15: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/15.jpg)
Log Shipping
![Page 16: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/16.jpg)
● Lightweight log shipper● Written in GO● Minimal resource usage● SSL● Transformation capabilities
Log Courier
![Page 17: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/17.jpg)
{ "general": { "log file": "/var/log/log-courier.log", "admin enabled": true }, "network": { "transport": "tls", "servers": [ "your.logstash.server:55516" ], "ssl certificate": "/var/lib/puppet/ssl/certs/yourcert.pem", "ssl key": "/var/lib/puppet/ssl/private_keys/yourkey.pem", "ssl ca": "/var/lib/puppet/ssl/certs/ca.pem", "timeout": 40 }, "files": [ { "paths": [ "/var/log/syslog" ], "fields": { "shipper": "log-courier", "type": "syslog" } },]
![Page 18: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/18.jpg)
input { courier { port => 55516 ssl_verify => true ssl_verify_ca => "/var/lib/puppet/ssl/certs/ca.pem" ssl_certificate => "/var/lib/puppet/ssl/certs/yourcert.pem" ssl_key => "/var/lib/puppet/ssl/private_keys/yourkey.pem" type => "log-courier" }}
![Page 19: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/19.jpg)
filter { if [type] == "syslog" { if [message] =~ /Registrar received .* event/ { drop {} } grok { match => [ "message", "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ] match => [ "message", "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY} %{TIME} %{POSINT:syslog_pid} %{WORD:severity} %{GREEDYDATA:syslog_message}"] match => [ "message", "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY} %{TIME} %{POSINT:syslog_pid} %{WORD:severity} %{GREEDYDATA:syslog_message}"] add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] add_field => [ "program", "%{syslog_program}" ] add_field => [ "timestamp", "%{syslog_timestamp}" ]
} syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } }}
![Page 20: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/20.jpg)
filter { if [type] == "native_syslog" { grok { match => [ "message", "%{SYSLOGLINE}" ] add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } }}
![Page 21: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/21.jpg)
filter {# Add in group tags we didn't add in forwarder due to bug# https://github.com/elasticsearch/logstash-forwarder/issues/65# By grouping the logs using tags we can then search all the related logs in kibana if [type] =~ /cinder.*/ { mutate { add_tag => [ "cinder", "oslofmt" ] } }}
![Page 22: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/22.jpg)
output { elasticsearch { host => elasticsearch embedded => false protocol => http }}output { if [type] == "syslog" { riemann { riemann_event => { "description" => "%{syslog_message}" "service" => "%{syslog_program}" "state" => "%{syslog_severity_code}" } } }}
![Page 23: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/23.jpg)
FILTER
aggregatealteranonymizecollatecsvcidrclonecipherchecksumdatede_dotdnsdropelasticsearchextractnumbersenvironmentelapsedfingerprintgeoipgroki18njsonjson_encodekvmutatemetricsmultilinemetaeventprunepunctrubyrangesyslog_prisleepsplitthrottletranslateuuidurldecodeuseragentxmlzeromq
INPUT
beatscouchdb_changesdrupal_dblogelasticsearchexeceventlogfilegangliagelfgeneratorgraphitegithubheartbeatherokuhttphttp_pollerircimapjdbcjmxkafkalog4jlumberjackmeetuppipepuppet_facterrelprssrackspacerabbitmqredissalesforcesnmptrapstdinsqlites3sqsstompsyslogtcptwitterunixudpvarnishlogwmiwebsocketxmppzenosszeromq
OUTPUT
boundarycirconuscsvcloudwatchdatadogdatadog_metricsemailelasticsearchelasticsearch_javaexecfilegoogle_bigquerygoogle_cloud_storagegangliagelfgraphtasticgraphitehipchathttpircinfluxdbjuggernautjirakafkalumberjacklibratologglymongodbmetriccatchernagiosnullnagios_nscaopentsdbpagerdutypiperiemannredminerackspacerabbitmqredisriaks3sqsstompstatsdsolr_httpsnssyslogstdouttcpudpwebhdfswebsocketxmppzabbixzeromq
![Page 24: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/24.jpg)
![Page 25: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/25.jpg)
![Page 26: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/26.jpg)
Riemann - an event stream processor● very low latency● extensive Clojure API● API can also be extended with Java
![Page 27: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/27.jpg)
(streams (where (and (service #"^riak") (state "critical")) (email "[email protected]")))
![Page 28: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/28.jpg)
(by [:host :service])
![Page 29: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/29.jpg)
(by [:host :service] (changed :state (rollup 5 3600 (email "[email protected]"))))
![Page 30: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/30.jpg)
(use 'clojure.java.io)
(defn get_messages [filename] (with-open [rdr (reader filename)] (doall (line-seq rdr))))
(def messages (get_messages "/etc/riemann.conf.d/riemann.whitelist"))
(def whitelist_pattern (str "^((?!(" (clojure.string/join "|" messages) ")).)*$"))
(def email(mailer { :from "[email protected]" }))
(streams (by :service (where (or (state "2")(state "1")(state "0")) (where (description (re-pattern whitelist_pattern)) (rollup 3 3600 (email "[email protected]" ))))))
![Page 31: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/31.jpg)
Ignoring invalid UTF-8 byte sequences in data to be sent to PuppetDBtftp: client does not accept optionsDHCP packet received on [a-zA-Z0-9-_]+ which has no addressCan\'t create new lease file: Permission denied\[\-\] Authorization failed\. The request you have made requires authentication\. from 127\.0\.0\.1\[\-\] \[instance: [a-zA-Z0-9-]+\] Instance not resizing[,] skipping migration\.^.*dhcp-failover rejected: incoming update is less critical than outgoing update$^.*Please use the the default quota class for default quota.$^.*FAILED: Has an address record but no DHCID, not mine.$^.*Found \d+ in the database and \d+ on the hypervisor.$^.*Arguments dropped when creating context.*^.*Failed to inspect.*of instance.*domain is in state of SHUTOFF^.*Unknown base file: /var/lib/nova/instances/_base/*^.*Couldn\'t obtain IP address of instance.*\[*\] IPMI message handler: BMC returned incorrect response, expected*\[-\] While synchronizing instance power states, found \d+ instances in the database and \d+ instances on the hypervisor
![Page 32: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/32.jpg)
(use 'clojure.java.io)
(defn get_messages [filename] (with-open [rdr (reader filename)] (doall (line-seq rdr))))
(def messages (get_messages "/etc/riemann.conf.d/riemann.blacklist"))
(def blacklist_pattern (str "^?(" (clojure.string/join "|" messages) ").*$"))
(def pd (pagerduty "pagerduty_api_key"))
(streams (by :host (where (description (re-pattern blacklist_pattern)) (with {:state "Failure" :service "Hardware"} (throttle 1 43200 #(info %) (:trigger pd))))))
![Page 33: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/33.jpg)
EDAC MC\d+: \d+ CE error on CPU#\d+Channel#\d+_DIMM#\d+.*ata\d+.\d+: exception.*ata\d+.\d+: failed command:.*ata\d+: link is slow to respond, please be patient.*ata\d+.\d+:.*failed.*
![Page 34: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/34.jpg)
Log files
log courier
logstash
elasticsearch
riemann
kibana
pagerduty
![Page 35: Matt Jarvis - Unravelling Logs: Log Processing with Logstash and Riemann](https://reader031.fdocuments.in/reader031/viewer/2022030304/5877061d1a28ab890e8b56d7/html5/thumbnails/35.jpg)
Thanks for Listening !