Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ireland 2015, Lightning...
-
Upload
brian-brazil -
Category
Internet
-
view
1.148 -
download
1
Transcript of Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ireland 2015, Lightning...
Better Monitoring for PythonInclusive Monitoring with Prometheus
Monitoring is often a Problem, not a SolutionThemes common among companies I talk to:
● Monitoring tools are limited, both technically and conceptually● Tools don’t scale well and are unwieldy to manage● Operational practices don’t align with the business
Result: Engineers continuously woken up for non-issues, get fatigued
Fundamental Challenge is Limited Visibility
PrometheusInspired by Google’s Borgmon monitoring system.
Started in 2012 by ex-Googlers working in Soundcloud as an open source project.
Mainly written in Go. Publically launched in early 2015.
100+ companies using it including Digital Ocean, GoPro, Apple, Red Hat and Google.
Services have Internals
Monitor the Internals
Monitor as a Service, not as Machines
Inclusive MonitoringDon’t monitor just at the edges:
● Instrument client libraries● Instrument server libraries (e.g. HTTP/RPC)● Instrument business logic
Library authors get information about usage.
Application developers get monitoring of common components for free.
Dashboards and alerting can be provided out of the box, customised for your organisation!
Let’s Talk Codepip install prometheus_client
from prometheus_client import Summary, start_http_serverREQUEST_DURATION = Summary('request_duration_seconds', 'Request duration in seconds')
@REQUEST_DURATION.time()def my_handler(request): pass // Your code here
start_http_server(8000)
Multiple Dimensions (No Evil Twins Please)from prometheus_client import CounterREQUESTS = Summary('requests_total', 'Total requests', ['method'])
def my_handler(request): REQUESTS.labels(request.method).inc() pass // Your code here
Exceptional Circumstances In Progressfrom prometheus_client import Counter, GaugeEXCEPTIONS = Counter('exceptions_total', 'Total exceptions')IN_PROGRESS = Gauge('inprogress_requests', 'In progress')
@EXCEPTIONS.count_exceptions()@IN_PROGRESS.track_inprogress()def my_handler(request): pass // Your code here
Getting Data Outfrom prometheus_client import start_http_server
if __name__ == '__main__': start_http_server(8080)
Also possible with Django, Twisted etc.
Oh Noes, My Vendor Lock In!from prometheus_client.bridge.graphite import GraphiteBridgegb = GraphiteBridge(('graphite.your.org', 2003))
gb.start(10.0) # Push every 10 seconds
Exposition to Prometheus doesn’t use a special API, can be hooked into whatever monitoring system you have.
Instrument once, work with everything!
Running PrometheusGrab binary from http://www.robustperception.io/prometheus-nightly-binaries/
Put config in prometheus.yml:
scrape_configs: - job_name: python target_groups: targets: [localhost:8000]Run it:
./prometheus
Dashboards
Prometheus is as Powerful as PythonCan slide, dice, aggregate and do math to produce your alerts
If it’s computable, Prometheus can compute it*
Alert on what matters, not on what your monitoring limit you to!
* Halting, tractability and sanity not guaranteed
Links!
Blog Posts: www.robustperception.io/tag/prometheus
Demo: demo.robustperception.io
Python Client: https://github.com/prometheus/client_python
Project Website: prometheus.io