BLEK at section.ioAnd some other bits

•Open•Easy•Give the user control

The Three Tenets

•Experience•Community•Approachable

Choosing Elastic

•Syslog•Web access logs•Proxy error logs•Docker container metrics•Provisioning and deployment logs•User events•API calls

All the logs

Log flow

Delivery

networks

Logstash

receivers

redis

Logstash processor

s

Logstash senders

redis

Ops Elasticsearch

clusterApps

Elasticsearch

cluster

StatsD, Carbon

Optimising to reduce latency here

Current scale

•About 150,000 log entries received per minute•About 150 million documents indexed per day• Just over 1 billion documents in the Apps ES cluster•One index per day for the last 7 days

Why StatsD, Carbon, and Graphite?

•Optimisation of frequent queries• Response times• Error rates• Cache hit/miss ratio

•Longer retention of key metrics:• 1 minute granularity for 1 month• 1 hour granularity for 13 months

•syslog ⇒ logstash-forwarder ⇒ Filebeat•Filebeat restarts•Packetbeat TLS

Beats

•Elasticsearch API filtering• Just-in-time Kibana containers

Fun with proxies

Jason Stangroome

Twitter: @jstangroomehttps://blog.stangroome.comhttps://www.section.io/blog

Thank you