Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data...
Transcript of Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data...
![Page 1: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/1.jpg)
Prometheus:A Next-Generation Monitoring System
Björn Rabenstein, Julius VolzSoundCloud
SREcon DublinMay 14, 2015
![Page 2: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/2.jpg)
History
![Page 3: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/3.jpg)
From a monolith...
![Page 4: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/4.jpg)
...to microservices
![Page 5: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/5.jpg)
Existing monitoring setup
in 2012...
![Page 6: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/6.jpg)
Service monitoring
![Page 7: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/7.jpg)
Service monitoring
![Page 8: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/8.jpg)
Host monitoring
![Page 9: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/9.jpg)
Alerting
![Page 10: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/10.jpg)
Not fun
![Page 11: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/11.jpg)
Prometheus
![Page 12: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/12.jpg)
SOUNDCLOUD
Multi-dimensional data model
api_http_requests_total{method="GET", endpoint="/api/tracks", status="200"} 2034834
(like OpenTSDB)
![Page 13: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/13.jpg)
SOUNDCLOUD
Operational simplicity
$ go build$ ./prometheus
(not like OpenTSDB)
![Page 14: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/14.jpg)
SOUNDCLOUD
Scalable data collection
Thousands of targets. Hundreds of thousands of samples per second.
Millions of time series.On a single monitoring server.
Running many servers is easy, too…Pull, not push.
![Page 15: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/15.jpg)
SOUNDCLOUD
Powerful query language
topk(3, sum(rate(bazooka_instance_cpu_time_ns[5m])) by (app, proc))
sort_desc(sum(bazooka_instance_memory_limit_bytes - bazooka_instance_memory_usage_bytes) by (app, proc))
![Page 16: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/16.jpg)
SOUNDCLOUD
Architecture
![Page 17: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/17.jpg)
SOUNDCLOUD
Expression browser
![Page 18: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/18.jpg)
SOUNDCLOUD
Built-in graphing
![Page 19: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/19.jpg)
SOUNDCLOUD
PromDash
![Page 20: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/20.jpg)
Alertmanager
![Page 21: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/21.jpg)
Exporters
vs.
direct instrumentation
![Page 22: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/22.jpg)
Client libraries
Official
GoJava (JVM)RubyPython
Unofficial
.NET / C#Node.jsBash
![Page 23: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/23.jpg)
Integrations
Official exporters
Node/system metrics exporterGraphite exporterCollectd exporterJMX exporterHAProxy exporterStatsD bridgeAWS CloudWatch exporterHystrix metrics publisherMesos task exporterConsul exporter
Unofficial exporters
RethinkDB exporterRedis exporterscollector exporterMongoDB exporterDjango exporterGoogle's mtail log data extractorMinecraft exporter moduleMeteor JS web framework exporter
Direct instrumentation
cAdvisorKubernetesKubernetes-MesosEtcdgokitgo-metrics instrumentation library...
![Page 24: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/24.jpg)
SOUNDCLOUD
Further reading
prometheus.io
developers.soundcloud.com/blog
boxever.com/tech-blog
![Page 25: Prometheus - USENIX · PDF file$ ./prometheus (not like OpenTSDB) SOUNDCLOUD Scalable data collection Thousands of targets. Hundreds of thousands of samples per](https://reader034.fdocuments.in/reader034/viewer/2022051405/5a9f97717f8b9a7f178d032a/html5/thumbnails/25.jpg)
SOUNDCLOUD
AcknowledgementsMany contributors at SoundCloud…
...and by now so many more from everywhere.
Special thanks to:
Matt T. Proud (founding father)Brian Brazil (Boxever)
Johannes “fish” Ziemke (Docker)