Monitoring Cassandra with Riemann

download Monitoring Cassandra with Riemann

of 46

  • date post

    27-Aug-2014
  • Category

    Software

  • view

    365
  • download

    4

Embed Size (px)

description

Monitor your Cassandra cluster using Riemann

Transcript of Monitoring Cassandra with Riemann

  • Real-time Monitoring with Riemann Patricia Gorla @patriciagorla Cassandra Consultant www.thelastpickle.com OSCON
  • About Us Work with clients to deliver and improve Apache Cassandra services Apache Cassandra committer, Datastax MVP, Hector maintainer, Apache Usergrid committer Based in New Zealand & USA
  • Overview Setting up Riemann Simple Monitoring Aggregate Monitoring
  • Monitoring a cluster
  • database Compactions Pending! Data Size! Read/Write Requests! Read/Write Latency! CF SSTable Size
  • database Compactions Pending! Data Size! Read/Write Requests! Read/Write Latency! CF SSTable Size Read/Write Request Timeouts! Dropped Messages! Hints Stored networking
  • database Compactions Pending! Data Size! Read/Write Requests! Read/Write Latency! CF SSTable Size Read/Write Request Timeouts! Dropped Messages! Hints Stored networking JVM CMS Collection Time! ParNew Collection Time
  • org.apache.cassandra.metrics Codahale metrics library rst introduced in CASSANDRA-3671 (v 1.1) Pluggable reporters introduced in CASSANDRA-4430 (v 2.0.2) Complete description in Apache documentation Dropped Messages Cache Commit Log Compaction Connection Client Request Storage
  • Riemann Real-time copyright Kyle Kingsbury, used with permission
  • Riemann Real-time Streams are successions of lters copyright Kyle Kingsbury, used with permission
  • Riemann Real-time Streams are successions of lters Expressive programming language copyright Kyle Kingsbury, used with permission
  • Monitoring Ecosystem
  • Download package from riemann.io
  • /etc/riemann/riemann.cong
  • Modied version of https://github.com/addthis/metrics-reporter-cong
  • Specify the metrics to send from Cassandra
  • cassandra-env.sh
  • Starting Cassandra
  • INFO [2014-07-15 21:44:50,712] pool-1-thread-1 - riemann.config - average event #riemann.codec.Event{ :host x.x.x.x, :service org.apache.cassandra.metrics ClientRequest Write Latency .95, :state ok, :description nil, :metric 19514.037499999988, :tags nil, :time 1405482287, :ttl nil }
  • INFO [2014-07-15 21:44:50,712] pool-1-thread-1 - riemann.config - average event #riemann.codec.Event{ :host x.x.x.x, :service org.apache.cassandra.metrics ClientRequest Write Latency .95, :state ok, :description nil, :metric 19514.037499999988, :tags nil, :time 1405482287, :ttl nil }
  • INFO [2014-07-15 21:44:50,712] pool-1-thread-1 - riemann.config - average event #riemann.codec.Event{ :host x.x.x.x, :service org.apache.cassandra.metrics ClientRequest Write Latency .95, :state ok, :description nil, :metric 19514.037499999988, :tags nil, :time 1405482287, :ttl nil }
  • INFO [2014-07-15 21:44:50,712] pool-1-thread-1 - riemann.config - average event #riemann.codec.Event{ :host x.x.x.x, :service org.apache.cassandra.metrics ClientRequest Write Latency .95, :state ok, :description nil, :metric 19514.037499999988, :tags nil, :time 1405482287, :ttl nil }
  • INFO [2014-07-15 21:44:50,712] pool-1-thread-1 - riemann.config - average event #riemann.codec.Event{ :host x.x.x.x, :service org.apache.cassandra.metrics ClientRequest Write Latency .95, :state ok, :description nil, :metric 19514.037499999988, :tags nil, :time 1405482287, :ttl nil } package name class module
  • INFO [2014-07-15 21:44:50,712] pool-1-thread-1 - riemann.config - average event #riemann.codec.Event{ :host x.x.x.x, :service org.apache.cassandra.metrics ClientRequest Write Latency .95, :state ok, :description nil, :metric 19514.037499999988, :tags nil, :time 1405482287, :ttl nil }
  • /etc/riemann/riemann.cong
  • Flexible Simple to set up Keyboard shortcuts riemann-dash
  • Edit graphs with keyboard shortcuts
  • Mean 95th Percentile Latency across all nodes
  • Mean 95th Percentile Latency across all nodes
  • Can also use xed-event-window for less regular occurrences
  • Mean 95th Percentile Latency across all nodes ++
  • Mean 95th Percentile Latency across all nodes ++
  • Mean 95th Percentile Latency across all nodes ++
  • (avg) Chart in riemann-dash
  • (avg) Chart in riemann-dash
  • (avg) Detecting threshold breaches
  • Generalise which metrics to monitor
  • Call the custom function in the (streams ) section
  • Patricia Gorla @patriciagorla Cassandra Consultant www.thelastpickle.com OSCON Q&A