Real-time Application monitoring

Amit Gupta

Central App Monitoring Platform

• Graphite– Carbon– Whisper– Webapp

• Simple time series data• Highly scalable, optimized for I/O• Queuing/Caching• Capability to club non-persisted metrics with persisted while

returning data for monitoring (Real-time)• Carbon-relay -> Clustering• Works fine for lacs of metrics coming every second• Can work with any client (statd daemon, jmxtrans, yammer etc.

or we can write our own easily) because of simple spec

Central App Monitoring Platform• Metrics names recommended to be given in reverse dns format

– Easy to aggregate metrics for multiple servers of a system (eg. com.snapdeal.cams.*)

• Supports both TCP & UDP• Lots of mathematical functions to derive, aggregate, join multiple

metrics• Grafana used for visualization (though graphite also provides UI)• Grafana

– Cutting edge graphs and very easy configuration through GUI– Easily integrates with Graphite APIs– Integrated with elastic search for faster response on graphs– Many more calculations (average, max, min etc.) possible on UI itself

Application Metrics – Yammer• Simple Java library• Different kind of Metrics

– Gauge– Counter– Meter– Histogram– Timer– HealthCheck

• Supports both push (configurable frequency) or pull (jmx or http)• Also sends JVM Info by default• Very less overhead on application

– Just instantiate POJOs in between your existing code– All processing in app’s control– No need to write separate code– Pushing via separate thread pool (no impact on app even if server is down)

• Yammer metrics can be exposed as MBeans (for JMX) as well.

Application Metrics – Jmxtrans

• What if you want to monitor your existing Mbeans via central monitoring platform (like graphite) or there are 3rd party frameworks already exposing data using JMX

• Use Jmxtrans to integrate with Graphite server• Frequency of pushing can be configured• A separate centralized Jmxtrans App can suffice

for multiple applications (though running as agent in same process also possible)

Alerting - Nagios

• Little crude, but flexible & powerful way to alert.

• Client/Server Architecture

Architecture Diagram

Thank You