LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic...
Transcript of LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic...
![Page 1: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/1.jpg)
LOG AGGREGATION
To better manage your Red Hat footprint
Miguel Pérez ColinoStrategic Design Team - ISBU2017-05-03
@mmmmmmpc
![Page 2: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/2.jpg)
AgendaManaging your Red Hat footprint with Log Aggregation
● The Situation● The Challenge ● The Solution
![Page 3: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/3.jpg)
THE SITUATION
![Page 4: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/4.jpg)
Cloud DeploymentsThey do really scale ...
https://www.cncf.io/blog/2016/08/23/deploying-1000-nodes-of-openshift-on-the-cncf-cluster-part-1/
● Higher scalability● More workloads per physical
machine (multi-tenant)● Network and Storage also
Software Defined● Containers and Microservices
providing more granularity
![Page 5: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/5.jpg)
Cloud DeploymentsAct as one single thing …
… and need to be managed and operated as one
Source: https://commons.wikimedia.org/wiki/File:Auklet_flock_Shumagins_1986.jpg
![Page 6: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/6.jpg)
THE CHALLENGE
![Page 7: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/7.jpg)
Data (What)Data + Information flow in Log Aggregation
ProcessIngest StoreCollect Query ViewGenerate
Derived from: http://www.dataintensive.info/
![Page 8: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/8.jpg)
Personas (Who)That can use Log Aggregation
Log Aggregation
MonitoringProvides Events, Consumes Logs
Cloud OpsRoot Cause Analysis
DeveloperApp Analysis & Debug
Security EngineerSec Analysis, Audits
User / MarketingAccess to stats
Service DesignerIT Manager
Access to aggregated data, i.e. SLA, usage
![Page 9: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/9.jpg)
Personas (Motivation)That need Log Aggregation
Cloud Ops (Apps)
“I want to proactively know about active or potential degradation of service”
Cloud Ops (OpenStack)
“User reports that their VM request failed and returned error”
Developer (OpenShift)
“My recent commit resulted in Jenkins test failure”
“Application (multi-tiered) launched from CloudForms returns error”
Cloud Suite User
![Page 10: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/10.jpg)
Situational Awareness (Why)Or the need of it!
Source: https://en.wikipedia.org/wiki/Situation_awareness
![Page 11: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/11.jpg)
THE SOLUTION
![Page 12: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/12.jpg)
ArchitectureProposed General Architecture
Real TimeAnalytics
and Response
Host
Bus
N N N
Archive
Data Store
GeneralVisualization
MC
MC
Storage
Legend
MC
N
Message Client
Normalizer
C
C
C Collector
Slide Credit: Tushar Katarki [@tkatarki]
![Page 13: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/13.jpg)
ImplementationIntroduction to EFK
Kibana ElasticSearch Log Source Fluentd
User Interface for:● Search● Graph● Dashboard
Index and store data and metadata
making search fast and reliable
● Parsing● Filtering● Enriching● Deleting● Output
Buffering
● TCP/UDP● HTTP● File: Text● Stdout: CSV,
JSON, MessagePack
● syslog/journal
Slide Credit: Tushar Katarki [@tkatarki]
![Page 14: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/14.jpg)
Current StatusBeing delivered and supported
OpenShift Container Platform 3.5
● Full EFK stack provided as containers
OpenStack Platform 10
● Fluentd as log collector
Red Hat Virtualization
● Coming Soon!
Log files
Journal Fluentd
KuberentesServices
Syslog
Master Nodes
Elasticsearch
Kibana
...Application Nodes
Log files
Journal Fluentd
App inside container
Syslog
Infra Nodes
Elasticsearch
Kibana
host logs
App inside container
Elasticsearch
Curator
Multi-Tenant Access
Diagram Credit: Tushar Katarki [@tkatarki]
![Page 15: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/15.jpg)
BEYOND ...
![Page 16: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/16.jpg)
Common Data ModelTo ensure integration and interoperability
What Is It?
● A Data Model for Logs (and other data) to identify and tag data (i.e. log fields)
Why?
● Alignment/Correlation with different RH products● Improved maintainability of Data● Better presentation/data consumption● Enables 3rd party ecosystem● Facilitates deep learning analysis of data
Ingestion pipeline
Consumption pipeline
Indexing and Storage
![Page 17: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/17.jpg)
Common Data ModelExample ...
Data extracted:
● Container name● Pod name● Namespace name● Docker container ID
K8S data queried:
● Pod UID● Pod labels● Pod host● Namespace UID.
All merged into output log in JSON Format
Images Credit: Anton Sherkhonov [@peatz]
CDM
A → 1B → 2C → 3
![Page 18: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/18.jpg)
User ExperiencePrototyping and validating dashboards for users
Slide Credits: Peter Portante & Vince Conzola
![Page 19: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/19.jpg)
Exploring different approachesPrototyping with alternative toolsets with partners
Slide Credits: Luca Rosellini (Keedio)
![Page 20: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/20.jpg)
ACTION!
![Page 21: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/21.jpg)
How are you doing it?Please, provide your feedback ...
http://bit.ly/log-aggregation
![Page 22: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/22.jpg)
THANK YOUplus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews
![Page 23: LOG AGGREGATION - Red Hat...To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f225bc02f3cb303aa011684/html5/thumbnails/23.jpg)