Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH...
Transcript of Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH...
![Page 1: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/1.jpg)
1
Aravind PutrevuDeveloper | Evangelist@aravindputrevu | aravindputrevu.in
Elastic StackMonitor your Services
![Page 2: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/2.jpg)
22
Agenda
Why Monitoring?1
Beats : Lightweight data shipper framework3
Monitor All things with Beats4
Demo5
Why Elastic Stack?2
![Page 3: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/3.jpg)
33
Agenda
Why Monitoring?1
Beats : Lightweight data shipper framework3
Monitor All things with Beats4
Demo5
Why Elastic Stack?2
![Page 4: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/4.jpg)
44
Agenda
Why Monitoring?1
Beats : Lightweight data shipper framework3
Monitor All things with Beats4
Demo5
Why Elastic Stack?2
![Page 5: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/5.jpg)
55
Agenda
Why Monitoring?1
Beats : Lightweight data shipper framework3
Monitor All things with Beats4
Demo5
Why Elastic Stack?2
![Page 6: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/6.jpg)
66
Agenda
Why Monitoring?1
Beats : Lightweight data shipper framework3
Monitor All things with Beats4
Demo5
Why Elastic Stack?2
![Page 7: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/7.jpg)
7
Elastic StackNo enterprise edition
All new versions with 6.3
X-Pack
Security
Alerting
Monitoring
Reporting
Machine Learning
Graph
![Page 8: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/8.jpg)
8
Why Monitoring?Pet vs Cattle
![Page 9: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/9.jpg)
9
Why Monitoring?Find out what’s happening?
![Page 10: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/10.jpg)
10
Why Monitoring?Resolving Errors and bottlenecks
![Page 11: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/11.jpg)
11
\
LOGANALYTICS
METRICS ANALYTICS
BUSINESS ANALYTICS SEARCHSECURITY
ANALYTICS
Monitor your Elastic Stack
Find links in your data
Be alerted on changes
Protect your data
Share your insights
Detect anomalies
APM
Why Elastic?
![Page 12: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/12.jpg)
Beats
Log Files Metrics
Wire Data
Datastore Web APIs
Social Sensors
Kafka
Redis
MessagingQueue
ES-Hadoop
Elasticsearch
Kibana
Master Nodes (3)
Ingest Nodes (X)
Data Nodes – Hot (X)
Data Notes – Warm (X)
Instances (X)
your{beat}
X-Pack X-Pack
Custom UI
LDAP
Authentication
AD
Notification
SSO
Hadoop Ecosystem
Logstash
Nodes (X)
![Page 13: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/13.jpg)
Logstash
Nodes (X)
Datastore Web APIs
Social Sensors
Kafka
Redis
MessagingQueue
ES-Hadoop
Elasticsearch
Kibana
Master Nodes (3)
Ingest Nodes (X)
Data Nodes – Hot (X)
Data Notes – Warm (X)
Instances (X)
X-Pack X-Pack
Custom UI
LDAP
Authentication
AD
Notification
SSO
Hadoop Ecosystem
Beats
Log Files Metrics
Wire Data your{beat}
![Page 14: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/14.jpg)
14
BeatsLightweight data shippers
Ship data from the sourceShip and centralize in
ElasticsearchShip to Logstash for
transformation and parsing
Ship to Elastic Cloud Libbeat: API framework to build custom beats 30+ community Beats
![Page 15: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/15.jpg)
15
The Beats family
HeartbeatUptime monitoring
FilebeatLog files
WinlogbeatWindows Event Logs
PacketbeatNetwork data
+40 community
BeatsMetricbeat
Metrics
AuditbeatAudit data
Apachebeat, dockbeat, httpbeat,
mysqlbeat, nginxbeat, redis
beats, twitterbeat, and more
{your}beat
![Page 16: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/16.jpg)
Logstash vs Beats
16
● Beats are lightweight data shippers that you install as agents on your servers
● Logstash has a larger footprint, but provides a broad array of input, filter, and output plugins for collecting, enriching, and transforming data from a variety of sources.
![Page 17: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/17.jpg)
How beats work?
17
Log Files Metrics
Wire Data your{beat}
Beats Framework
libbeat
• Small application
• Install as agent on your servers
• Written in Golang
• No runtime dependencies
• Single purpose
![Page 18: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/18.jpg)
How beats work?
18
![Page 19: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/19.jpg)
19
Classic Deployments
VM 1
Metricbeat
Filebeat
VM 2
Metricbeat
Filebeat
VM n
Metricbeat
Filebeat
![Page 20: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/20.jpg)
20
Kubernetes deployment
Node 1 Node 2 Node n
?
![Page 21: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/21.jpg)
21
CENTRALIZED COLLECTION
Logstash
Elasticsearch
Transform
Store
ingest node
data node
network devices
DISTRIBUTED COLLECTION
Beats
servers, containers
Elastic evolving ingest story
![Page 22: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/22.jpg)
22
Immediate insights with modules
• Turnkey experience for specific data types
• Data to dashboard in just one step
• Automated parsing and enrichment
• Default dashboards, alerts, ML jobs
Logging Metrics Security
Available with
![Page 23: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/23.jpg)
23
Logging modules
System• Linux / MacOS• Windows Events
Containers• Docker• Kubernetes
Infrastructure Applications
Databases• MySQL• PostgreSQL
Queues• Kafka• Redis
Web servers• Apache• Nginx
Audit data• Filesystem• System calls
WINLOGBEATFILEBEATAUDITBEAT
![Page 24: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/24.jpg)
24
Metrics modules
System• Linux• MacOS• Windows• Perfmon
Infrastructure
Cloud• AWS• Azure• DigitalOcean• GCP
Containers• Docker• Kubernetes
Virtualization• vSphere
PACKETBEATMETRICBEAT
Network• Netflow• Packets• TLS EnvelopeStorage• Ceph
LOGSTASH
![Page 25: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/25.jpg)
25
Metrics modules
ApplicationsDatastores• MySQL• PostgreSQL• MongoDB• Couchbase• Aerospike• Graphite
Web servers• Apache• Nginx
Other• HAProxy• Zookeeper
Queues• Kafka• Redis• RabbitMQ
Caches• Memcached
Uptime• Heartbeat
Custom apps• JMX/Jolokia• PHP-FPM• Golang
PACKETBEATMETRICBEAT LOGSTASHHEARTBEAT
![Page 26: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/26.jpg)
2626
We need specific tools to track things down
With containers architecture, everything is a moving target
![Page 27: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/27.jpg)
27
volume mounts volume mounts
Docker deployment
Kibana
Elasticsearch
docker host
Log files(/var/lib/docker/containers)
Web Apps Services
Docker API
...
Networking
FilebeatMetricbeat
/proc filesystem
![Page 28: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/28.jpg)
28
Kubernetes deployment
Node 1
Metricbeat
Filebeat
Node 2
Metricbeat
Filebeat
Node n
Metricbeat
Filebeat
Filebeat DaemonSet
Metricbeat DaemonSet
![Page 29: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/29.jpg)
29
Docker logs input
filebeat.prospectors: - type: docker containers.ids: - ‘*’
Parse and ship /var/lib/docker/containers/*/*.log:
{"log":"INFO elasticsearch/client.go:145 Elasticsearch url:http://elasticsearch:9200\r\n","stream":"stdout","time":"2018-02-11T23:29:19.236692181Z"}
Retrieve logs from Docker containers
![Page 30: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/30.jpg)
30
Metadata processorsEnrich events with useful metadata to correlate logs, metrics & traces
• cloud.region
• cloud.instance_id
• cloud.machine_type
• cloud.provider
• docker.container.id
• docker.container.image
• docker.container.name
• docker.container.labels
• kubernetes.pod.name
• kubernetes.namespace
• kubernetes.labels
• kubernetes.annotations
• kubernetes.container.name
• kubernetes.container.image
add_cloud_metadata add_docker_metadata add_kubernetes_metadata
![Page 31: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/31.jpg)
31
Metadata processorsExample{ "@timestamp": "2017-11-17T00:53:33.759Z", "message": "2017/11/07 00:53:32.804991 client.go:651: INFO Connected to Elasticsearch version 6.0.0", "kubernetes": { "pod": { "name": "filebeat-vqf85" }, "container": { "name": "filebeat" }, "namespace": "kube-system", "labels": { "k8s-app": "filebeat", "kubernetes.io/cluster-service": "true" } }, "meta": { "cloud": { "instance_id": "1234567", "provider": "digitalocean", "region": "blr1" } },}
![Page 32: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/32.jpg)
32
Metadata processorsadd_kubernetes_metadata internals
API Server
add_kubernetes_metadata
pod watcher
Pod start/stop events
Docker Logs
Cont. ID Metadata
update
418a913c7076 ………………c626cfdf38614 ………………e5563a7cb80e ………………73de79be045c ……………...
ElasticsearchParseEnrich
![Page 33: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/33.jpg)
33
Autodiscover
metricbeat.autodiscover: providers: - type: docker templates: - condition: contains.docker.container.image: etcd config: - module: etcd metricsets: ["leader", "self", "store"] hosts: "${data.host}:2379"
Watch Docker events and react to changes
![Page 34: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/34.jpg)
34
AutodiscoverWatch Docker events and react to changes
Events APIContainer start/stop events
Beats
{ "host": "10.4.15.9", "port": 2379, "docker": { "container": { "id": "13a2...d716" "name": "etcd", "image": "quay.io/coreos/etcd:v3.0.0", "labels": { "io.kubernetes.pod.name": "etcd-4dk4c", "io.kubernetes.pod.namespace": "kube-system" ... } } }}
1. autodiscover event
- module: etcd metricsets: ["leader", "self", "store"] hosts: "${data.host}:2379"
config template
3. var expansion
2. match condition
4. launch module
- module: etcd hosts: "10.4.15.9:237 metricsets: ["leader", "self", "store"]9"
![Page 35: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/35.jpg)
35
DEMO
![Page 36: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/36.jpg)
3636
What Next?
![Page 37: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/37.jpg)
37
How Elastic Stack can help you?
![Page 38: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/38.jpg)
38
● 100% Open Source
● Readymade UI in Kibana
● Language Agents
alpha
beta
![Page 39: Elastic Stack Monitor your Services · Elastic Stack Monitor your Services. 2 Agenda ... SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f6053888bcd982d795c5f5a/html5/thumbnails/39.jpg)
Resources
• https://www.elastic.co/learn• https://www.elastic.co/blog/category/engineering• https://discuss.elastic.co/• https://fb.com/groups/ElasticIndiaUserGroup• https://elastic.co/community
39