Latency: The Silent Monitoring System Killer

LATENCYTHE SILENT

MONITORING SYSTEM KILLER

Saturday, 21 January 2012

#MONITORINGSUCKS


SCALABILITY


execute large volumes of monitoring checks under a variety of conditions

(good + bad) with a consistent throughput


CONSISTENT THROUGHPUT


SHORT CHECK EXECUTION TIME


WHAT INTRODUCES VARIABILITY?


LATENCY INTRODUCED c SYNCHRONOUS CALLS


= 1 monitoring check


150 monitoring checkseach executed every 300 secondseach takes 1 secondchecks are executed serially


all checks executed in 150 secondsmonitoring system at 50% capacity


DOUBLE THE CHECKS


all checks executed in 300 secondsmonitoring system at 100% capacity


DOUBLE THE EXECUTION TIME


all checks executed in 600 secondsmonitoring system at 200% capacityonly 50% of the checks are “on time”


CHECK LATENCY


HOW DO WE FIX THIS!?


PERFORMANCE ANALYSIS!


AN ANALOGY


MONITORING CHECK ==

“ACTION” ON MVC WEB APP


SEPARATE DATA COLLECTION

FROM

THRESHOLDINGf NOTIFICATIONS


THIS SHIFTS LATENCY


IT DOES NOT ELIMINATE IT!


RRDTOOL IS EVIL


USE SOMETHING BETTER!


USE

OPENTSDBOR

GANGLIAWITH

CHECK_TSDBOR

CHECK_GMOND


A DIFFERENT SET a PROBLEMS


STORAGE WILL GO AWAY


CHAOS WILL ENSUE


PAGERS WILL MELT


SET UP “META-

PARENTING”


BUILD A KILL SWITCH


Technology