Graylog for open stack 3 steps to know why
-
Upload
viet-stack -
Category
Technology
-
view
203 -
download
3
Transcript of Graylog for open stack 3 steps to know why
MediTech JSChttps://meditech.vn
Private Cloud
Storage
Monitor
Logging
Managed Services
About me
Dinh Van Manh
● System Integration Department in MediTechJSC● Member of Hocchudong● Interested in OpenStack, Linux, Monitoring, Logging
and new technology● Habbit : “tra da + thuoc lao” with friends
Agenda
1. Log Overview1.1. Logs : What & Where?1.2. Why look at Logs1.3. How to use Logs effectively
2. Log in OpenStack2.1. OpenStack log statistics2.2. OpenStack Log Management : in imagionation & in fact
3. Graylog for OpenStack3.1. Introduce about Graylog3.2. Key features3.3. Architecture/Mechanism/Model of Graylog3.4. Graylog for OpenStack: 3 steps to know WHY?
4. Demo + Q.A
1.1. Logs : What & Where
What logs? (from the view of system administrator)
● System event diary
● System status records
● User activities
● Incident notify
Log format
1.1. Logs : What & Where
Log come from WHERE?
● Storage devices
● Application in Linux/Windows
● Cloud Services : OpenStack
● Servers
● Firewalls
● Routers, switches
1.2. Why look at Logs?
Basically :Incident response
higherTracking system event
higherMeasuring security : metrics, trends…
higher and higher Situational awarenessNew threat discoveryEstimating about user habit, trends...
1.3. How to use Logs effectively
Level 1 : Just SSH and view !● Understanding log location● Command to view log : tail, more, grep● Filtering by keyword
Level 2 : Use Syslog ● Collect syslog from client● Store in log server
Level 3 : Log management Software● Collect everything ● Retain most everything● Analyze enough● Summarize and report● Advance features : visualize, alert, share...
1.3. How to use Logs effectively
● Facility
○ Application Logs
○ Event Logs
○ Service Logs
○ System Logs
Log Keywords
● Severity○ 0 - emerg○ 1 - alert○ 2 - crit○ 3 - error○ 4 - warn○ 5 - notice○ 6 - info○ 7 - debug
● Rotention
○ Time to rotate log
● Retention
○ Delete, archive...log
● Syslog
○ protocol to transfer log
2.1. OpenStack log statistics
OpenStack System : 3 Controller + 30 Compute node ● Controller Node
○ 6 log folder per OpenStack service○ system log : auth, dmesg, kernel…○ application log : apache, haproxy,
pacemaker…
● Compute Node○ 2 log folder per OpenStack service○ system log : auth, dmes, kernel…○ application log : libvirt○ log of instances
=> Total : ● ~ 220 log file● 10 GB log = 30 million messages / day
2.2. OpenStack log management : in imagionation & in fact
Communication think Colleagues think In fact
When i said : My job is OpenStack log management !
So Waste !!! What should we do?
3.1. Graylog Introduce
● Log centralized management software
● Released in 2010 by Lenart Koopman with name is Graylog2
● In 1/2015 release Graylog v1., Graylog Inc was established
● Big change from Graylog version 2.0
● Newest version is Graylog 2.3.1, stable version is Graylog 2.3.0
3.2. Key features
Various Input & Output Analyze & Search
Visualize metricAlert & Trigger User management
3.3. Architecture/Mechanism/Model of Graylog
Overall architecture● Server
○ Graylog● Client
○ Client host○ Graylog sidecar○ Nxlog/Filebeat
Filebeat
Graylog Sidecar : Break the old path
● Configuration management system
● Config in client host only ONCE !
● All in Web● Secure with SSL/TLS
3.3. Architecture/Mechanism/Model of Graylog
Sidecar Work-flow : Easy config in 3 steps
Step 1 : Config in client● install sidecar● declare : graylog ip, client hostname, tags● start service
Step 2 : Config in Graylog Web● add tags● chose what logs you want to collect
Step 3 : Checking● Check colleted log
3.3. Architecture/Mechanism/Model of Graylog
Deep dive in architectureGraylog Server
● receive log message● execute log● communicate with other components
Elasticsearch ● store log message● search engine
MongoDB ● store meta infomation● store config data
3.3. Architecture/Mechanism/Model of Graylog
Log execute processingStep 1 :
● Spooling & store in disk temporarily● Prepare for buffer process
Step 2 : ● Messages from disk go in to Input Buffer● Mission : Filter, classify messages
Step 3 : ● Messages go in to Output Buffer● Onward to Elasticsearch or user defined
output
3.3. Architecture/Mechanism/Model of Graylog
Elasticsearch & Graylog
● Clustering
● Use API to communicate
● Use unicast-discovery to recogize other nodes
● Graylog as a Master Node
MongoDB & Graylog
● Client - Server mechanism
● Graylog use driver to communicate with MongoDB
Internal Graylog components mechanisms
3.4. Graylog for OpenStack : 3 steps to know WHY?
What should i do when instance spawning fail
A. Try to spawn again B. Blame for customer
D. Bug again! I’m quit ! C. Take a search in Graylog
Incident Response
Problem appear ! What should we do?
3.4. Graylog for OpenStack : 3 steps to know WHY?
Step 1 : Collect logsTake log from :
● nova log● neutron log● cinder log● glance log
Step 2 : AnalyzeMake a search in Graylog :Syntax : instance id + ERROR
Step 3 : Now you know WHYJust solve the problem & Go to sleep !
3.4. Graylog for OpenStack : 3 steps to know WHY?
Tracking a event
My instances was rebooted last night ??? When?