of Cable Tensions for Skyline Logging Systems - Oregon State
State of the art logging
-
Upload
balabit-it-security -
Category
Technology
-
view
325 -
download
3
Transcript of State of the art logging
![Page 1: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/1.jpg)
Copyright 2013 BalaBit IT Security Ltd.
State of the art loggingSyslog-ng, journal, CEE/Lumberjack and ELSA
Péter Czanikcommunity manager
![Page 2: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/2.jpg)
Copyright 2013 BalaBit IT Security Ltd.
Topics
• No, it is not about cutting trees :-)• What is syslog? And syslog-ng?• Free-form messages against name-value pairs• The new buzzword: journal• Standardization efforts: CEE/Lumberjack• Name-value pairs at work: ELSA
![Page 3: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/3.jpg)
Copyright 2013 BalaBit IT Security Ltd.
What is syslog?
• Logging: recording events
• Syslog:- Application: collecting events- Protocol: forwarding events
![Page 4: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/4.jpg)
Copyright 2013 BalaBit IT Security Ltd.
What is syslog-ng?
• “Next Generation” syslog server• “Swiss army knife” of logging
• More input sources (files, sockets, and so on)• Better filtering (not only priority, facility)• Processing (rewrite, normalize, correlate, and so
on)• More destinations (databases, encrypted network,
and so on)
![Page 5: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/5.jpg)
Copyright 2013 BalaBit IT Security Ltd.
What is new since 2.0
• 2.0 is best known, but EOL• Most important new features since 2.0:
- PatternDB and CSV message parsing- Correlation- SQL and MongoDB destinations- JSON formatting- Modularization- Multi-threading
• Next: 3.4- JSON parsing- More flexible configuration
![Page 6: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/6.jpg)
Copyright 2013 BalaBit IT Security Ltd.
Free form log messages
• Most logs are in /var/log• Most are from syslog (but also wtmp, apache, and
so on)• Most are: date + hostname + text
Mar 11 13:37:56 linux-6965 sshd[4547]: Accepted keyboard-interactive/pam for root from 127.0.0.1 port 46048 ssh2
• Text = English sentence with some variable parts• Easy to read
![Page 7: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/7.jpg)
Copyright 2013 BalaBit IT Security Ltd.
Why it does not scale?
• Few logs (workstation) → easy to find information• Many logs (server) → difficult to find information• Relevant information is presented differently by
each application• Difficult to process them with scripts
• Answer: structured logging- Events represented as name value pairs
![Page 8: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/8.jpg)
Copyright 2013 BalaBit IT Security Ltd.
Solution from syslog-ng: PatternDB
• Most messages are static texts with some variable parts embedded
• PatternDB parser:- Can extract useful information into name-value pairs- Add status fields based on message text
• Example:- user=root- action=login- status=failure
• It requires patterns• syslog-ng: name-value pairs inside
![Page 9: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/9.jpg)
Copyright 2013 BalaBit IT Security Ltd.
Journal
• The logging component of systemd• Name-value pairs inside:
- Message- Trusted properties- Any additional name-value pairs
• Native support for name-value pair storage
![Page 10: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/10.jpg)
Copyright 2013 BalaBit IT Security Ltd.
Journal: the enemy?
• FAQ: Q: is journal the enemy? A: No!• Journal is limited to Linux/systemd (syslog-ng: all
Linux/BSD/UNIX)• Journal is local only (syslog-ng: client – server)• Journal does not filter or process log messages
• Journal + syslog-ng complement each other• Logs forwarded to syslog-ng through:
/run/systemd/journal/syslog
• syslog-ng can filter, process and forward logs to many different destinations (one day also to journal)
![Page 11: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/11.jpg)
Copyright 2013 BalaBit IT Security Ltd.
CEE
• Journal, syslog-ng, Windows eventlog, rsyslog, auditd, and so on are based on name-value pairs
• All use different field names• Standardization is a must: CEE → Common Event
Expression• Events: name-value pairs instead of free-form text
- Taxonomy: name-value pairs to describe events (example: status)- Dictionary: name-value pairs for event parameters (example: user)
• PatternDB can turn free-form messages into CEE
![Page 12: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/12.jpg)
Copyright 2013 BalaBit IT Security Ltd.
Lumberjack
• Make CEE happen → implementation• Coordinated by RedHat
- CEE (Mitre), syslog-ng, rsyslog, and so on- Open, with high traffic mailing list- https://fedorahosted.org/lumberjack/
• API(s) to make structured logging easier• Work on dictionary, taxonomy, transport issues
![Page 13: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/13.jpg)
Copyright 2013 BalaBit IT Security Ltd.
Name-value pairs in action: ELSA
• ELSA: Enterprise Log Search and Archive• Based on syslog-ng, PatternDB and MySQL• Simple and powerful web GUI• Extreme scalability• Patterns focused on network security (Cisco,
Snort, HTTP, Bro, and so on)
![Page 14: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/14.jpg)
Copyright 2013 BalaBit IT Security Ltd.
Some logs
![Page 15: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/15.jpg)
Copyright 2013 BalaBit IT Security Ltd.
Diagram
![Page 16: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/16.jpg)
Copyright 2013 BalaBit IT Security Ltd.
A few extras
![Page 17: State of the art logging](https://reader034.fdocuments.in/reader034/viewer/2022042714/5554c839b4c90503388b5339/html5/thumbnails/17.jpg)
Copyright 2013 BalaBit IT Security Ltd.
Questions?
• Questions?