Centralised Logging with Logstash and · PDF fileCentralised Logging with Logstash and Kibana...
Transcript of Centralised Logging with Logstash and · PDF fileCentralised Logging with Logstash and Kibana...
![Page 1: Centralised Logging with Logstash and · PDF fileCentralised Logging with Logstash and Kibana (and rsyslog, and elasticsearch, and ...) Matthew Richardson (Engineering) 18th January](https://reader034.fdocuments.in/reader034/viewer/2022051203/5a9f94127f8b9a89178cec6f/html5/thumbnails/1.jpg)
Centralised Loggingwith
Logstash and Kibana
(and rsyslog, and elasticsearch,
and ...)
Matthew Richardson(Engineering)
18th January 2013
![Page 2: Centralised Logging with Logstash and · PDF fileCentralised Logging with Logstash and Kibana (and rsyslog, and elasticsearch, and ...) Matthew Richardson (Engineering) 18th January](https://reader034.fdocuments.in/reader034/viewer/2022051203/5a9f94127f8b9a89178cec6f/html5/thumbnails/2.jpg)
Centralised Logging?
![Page 3: Centralised Logging with Logstash and · PDF fileCentralised Logging with Logstash and Kibana (and rsyslog, and elasticsearch, and ...) Matthew Richardson (Engineering) 18th January](https://reader034.fdocuments.in/reader034/viewer/2022051203/5a9f94127f8b9a89178cec6f/html5/thumbnails/3.jpg)
Rsyslog
Default syslogd in LCFG
Easy to log centrally Disk Buffering for
safety
$ActionQueueType LinkedList
$ActionQueueFileName LogstashBuffer
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
$ActionQueueMaxDiskSpace 1G
*.* @@log.see.ed.ac.uk:5544RSYSLOG ?
![Page 4: Centralised Logging with Logstash and · PDF fileCentralised Logging with Logstash and Kibana (and rsyslog, and elasticsearch, and ...) Matthew Richardson (Engineering) 18th January](https://reader034.fdocuments.in/reader034/viewer/2022051203/5a9f94127f8b9a89178cec6f/html5/thumbnails/4.jpg)
Logstash
Inputs (25+)
Filters (20+)
Outputs (35+)
file grep filetcp/udp grok tcp/udpXMPP (Jabber)
mutate elasticsearch
log4j anonymize
XMPP (Jabber)
stdin dns emailWindows eventlog
XML ganglia
metrics graphite... ... ...
![Page 5: Centralised Logging with Logstash and · PDF fileCentralised Logging with Logstash and Kibana (and rsyslog, and elasticsearch, and ...) Matthew Richardson (Engineering) 18th January](https://reader034.fdocuments.in/reader034/viewer/2022051203/5a9f94127f8b9a89178cec6f/html5/thumbnails/5.jpg)
Logstash Configuration
TCP/UDP input Types and Tagging Powderful Filtering Structured Output
input { tcp { port => 5544 type => syslog }}filter { grok { pattern => [ "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{PROG:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ] } syslog_pri { type => "syslog" } date { match => [ “syslog_timestamp”, "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } mutate { exclude_tags => "_grokparsefailure" replace => [ "@message", "%{syslog_message}" ] } mutate { remove => [ "syslog_message", "syslog_timestamp" ] }}output { elasticsearch { embedded => false }}
RSYSLOGLOGSTASH
tcp input
LOGSTASHFilters ?
![Page 6: Centralised Logging with Logstash and · PDF fileCentralised Logging with Logstash and Kibana (and rsyslog, and elasticsearch, and ...) Matthew Richardson (Engineering) 18th January](https://reader034.fdocuments.in/reader034/viewer/2022051203/5a9f94127f8b9a89178cec6f/html5/thumbnails/6.jpg)
Log Analysis
Elasticsearch backend Kibana web frontend
ES/Kibana
LOGSTASHtcp input
LOGSTASHFilter → JSON
RSYSLOGLOGSTASH
tcp input
LOGSTASHFilters
![Page 7: Centralised Logging with Logstash and · PDF fileCentralised Logging with Logstash and Kibana (and rsyslog, and elasticsearch, and ...) Matthew Richardson (Engineering) 18th January](https://reader034.fdocuments.in/reader034/viewer/2022051203/5a9f94127f8b9a89178cec6f/html5/thumbnails/7.jpg)
Future Plans
Encryption/Authentication for logging Handling other (non-syslog) logs Other Inputs Metrics
ES/KibanaLOGSTASHFilter → JSON
RSYSLOGTLS
LOGSTASHtcp(tls) input
LOGSTASHFilters
LOGSTASHXMPP input
LOGSTASHUDP input
JabberServer
HardwareDevice
ApacheLCFG
etc
Graphite/Ganglia
XMPP/Email
Notification