Monitoring a Kubernetes-backed microservice architecture with ...
Transcript of Monitoring a Kubernetes-backed microservice architecture with ...
![Page 1: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/1.jpg)
Monitoring a Kubernetes-backed microservice architecture with Prometheus
Björn “Beorn” Rabenstein — SoundCloudFabian Reinartz — CoreOS
![Page 2: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/2.jpg)
SoundCloud 2012 – from monolith …
![Page 3: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/3.jpg)
… to microservices
![Page 4: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/4.jpg)
Orchestration neededRun containers in a cluster…
In-house innovation: Bazooka – PaaS, Heroko style.
Problems:
- Only 12-factor apps (stateless etc.).- Limited resource isolation.- No sidecars.- Maturity.
Meanwhile, the open-source world has evolved…
![Page 5: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/5.jpg)
K U B E R N E T E S
![Page 6: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/6.jpg)
Kubernetes- inspired by Google’s Borg
- not Borg
Today:
Tomorrow:
X
![Page 7: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/7.jpg)
LabelsLabels are the new hierarchies!
name = MyAPI
app = MyAPIversion = 0.4
app = MyAPIversion = 0.5
app = MyAPIversion = 0.4
app = MyDBcluster = auth
app = MyDBcluster = misc
select by [ app = MyAPI ]
service
podpod
podpod
pod
![Page 8: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/8.jpg)
Graphite
Monitoring at SC 2012
Service A
Service B
Service C
![Page 9: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/9.jpg)
Monitoring challenges ‐ A lot of traffic to monitor
- Monitoring traffic should not be proportional to user traffic
‐ Way more targets to monitor- One host can run many containers
‐ And they constantly change- Deploys, scaling, rescheduling unhealthy instances …
‐ Need a fleet-wide view.- What’s my overall 99th percentile latency?
‐ Still need to be able to drill down for troubleshooting- Which instance/endpoint/version/... causes those errors I’m seeing?
‐ Meaningful alerting- Symptom-based alerting for pages, cause-based alerting for warnings- See Rob Ewaschuk’s “My philosophy on alerting" https://goo.gl/2vrpSO
![Page 10: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/10.jpg)
Monitor everything, all levels, with the same system
Level What to monitor (examples) What exposes metrics (example)
Network Routers, switches SNMP exporter
Host (OS, hardware) Hardware failure, provisioning, host resources Node exporter
Container Resource usage, performance characteristics cAdvisor
Application Latency, errors, QPS, internal state Your own code
Orchestration Cluster resources, scheduling Kubernetes components
![Page 11: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/11.jpg)
P R O M E T H E U S
“Obviously the solution to all our problems with everything forever, right?”Benjamin Staffin, Fitbit Site Operations
![Page 12: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/12.jpg)
Prometheus- inspired by Google’s Borgmon
- not Borgmon
- initially developed at SoundCloud, open-source from the beginning
- public announcement early 2015
- collects metrics at scale via HTTP (think: yet another client of your microservice)
- thousands of targets, millions of time series, 800k samples/s, no dependencies
- easy to scale
![Page 13: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/13.jpg)
Features – multi-dimensional data model
http_requests_total{instance="web-1", path="/index", status="500", method="GET"}
http_requests_total{instance="web-1", path="/index", status="404", method="POST"}
http_requests_total{instance="web-3", path="/index", status="200", method="GET"}
#metrics x #values(instance) x #values(path) x #values(status) x #values(method)
▶ millions of time series
Labels are the new hierarchies!
![Page 14: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/14.jpg)
Features – powerful query language
The 3 path-method combinations with the highest number of failing requests?
topk(3, sum by(path, method) (rate(http_requests_total{status=~"5.."}[5m])
))
The 99th percentile request latency by request path?
histogram_quantile(0.99, sum by(le, path) (
rate(http_requests_duration_seconds_bucket[5m])
))
The questions to ask are often not known beforehand.
![Page 15: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/15.jpg)
Features – powerful query language
topk(3, sum by(path, method) (rate(http_requests_total{status=~"5.."}[5m])
))
{path="/api/comments", method="POST"} 105.4
{path="/api/user/:id", method="GET"} 34.122
{path="/api/comment/:id/edit", method="POST"} 29.31
![Page 16: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/16.jpg)
Features – easy instrumentationfrom prometheus_client import start_http_server, Histogram
# Create a metric to track time spent and requests made.
REQUEST_TIME = Histogram('request_processing_seconds', 'Time spent processing request')
# Decorate function with metric.
@REQUEST_TIME.time()
def process_request(t):
# do work … return
start_http_server(8000)
![Page 17: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/17.jpg)
Integrations (selection)
![Page 18: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/18.jpg)
K U B E R N E T E SP R O M E T H E U S
![Page 19: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/19.jpg)
![Page 20: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/20.jpg)
Three questions
How to monitor services running on Kubernetes with Prometheus?
X
?
?
How to monitor Kubernetes and containers with Prometheus?
How to run Prometheus on Kubernetes?
![Page 21: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/21.jpg)
Monitoring Services
svc = service-X
namespace = production
version = 22.4
version = 22.5
![Page 22: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/22.jpg)
Monitoring Services via Exporters
svc = mysql
namespace = production
MySQLexporter MySQL
![Page 23: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/23.jpg)
Monitoring Kubernetes
namespace = kube-system
apiserver
kubelet
kubelet
etcdkubelet
kubelet
kubeletkubelet
![Page 24: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/24.jpg)
Running Prometheus on Kubernetes
- So far: Prometheus ran outside of cluster- Pod IPs must be routable
- Conventional deployment (Chef, Puppet, …)
- Service discovery needs authentication
- To run Prometheus inside of cluster:
kubectl run --image="quay.io/prometheus/prometheus:0.18.0" prometheus
![Page 25: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/25.jpg)
Monitoring Services
svc = service-X
namespace = production
version = 22.4
version = 22.5
app = prometheus
![Page 26: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/26.jpg)
namespace = kube-system
apiserver
kubelet
kubelet
etcdkubelet
kubelet
kubeletkubelet
app = prometheus
namespace = infra
Monitoring Kubernetes
![Page 27: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/27.jpg)
What about storage?
A) None
B) Network/Cloud volumes
C) Host volumes
![Page 28: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/28.jpg)
DEMO
CC BY 2.0Author: carstingaxion / Carsten Bach
![Page 29: Monitoring a Kubernetes-backed microservice architecture with ...](https://reader034.fdocuments.in/reader034/viewer/2022051507/58a1ab6f1a28ab2a5d8bf247/html5/thumbnails/29.jpg)
The end