Post on 08-Jan-2017
Shipping & Visualize Your Data With ELK
Adam Chen
HELLO Everyone!I am AdamI am just an engineer like to share some experience with others.Thanks Gentoo let me familiar with Linux.
You can find me at @adaam
Let’s Start This Session
ELK? What !?
Develop by
Elastic Family
ELK ?DevOps ?
Log Still Play An Important Role
Today’s characters
Logstash/Beat seriesShipping all of your log to where it should go, like ES, AWS, or just text.
ElasticsearchThe main part to store your data with High Availability.
KibanaVisualize will power your data. To know more about its value.
Traditional Way to Collect Log
When error happened, administrator or RD/QA will need to login or write/use tool to grab log from each machines then analysis.
Hey Bob, Please collect the error log to analyze.
OK, Boss.
BOSS
Hey Bob, Please collect the error log to analyze.
Traditional Way to Collect Log
Old Way to Collect Log
How ELK Help ?
Centralize Log To One Place
Collect Log using ELK
Introduce The E, The L and The K
Logstash
An agent install at where log need to be collect.
Have much filter to process your log.Also Input/Output module.
Logstash Module
Logstash Filters
Date, geoip, json, kv ...etc
GROK !!
Grok online tester
Logstash DEMO
Input apache/nginx log then output to stdout, using local logstash.
Elasticsearch
SImply a data store with near real time search
Store data in index, can by hours, day, week, month. Setting at Logstash.
Machine spec depends on data size.
Elasticsearch Modules
They are improve ES function if you need.
Watcher, Shield, Marvel, Cloud-AWS
Elasticsearch DEMO
API Webpage
Kibana
Show your data to you understand.But you need to know your data fields first.
Kibana Demo
Create some template to show (data pregen at ES? )
Real time insert data to ES and see from template pregen. (apache/nginx?)
What else ?
Push Metrics to elsewhere
THANKS!Any questions?You can find me at:@adaam