Latency: The Silent Monitoring System Killer

Post on 19-May-2015

1.788 views 2 download

Tags:

description

Slides from a lighting talk presented at the January Sydney DevOps Meetup.

Transcript of Latency: The Silent Monitoring System Killer

LATENCYTHE SILENT

MONITORING SYSTEM KILLER

Saturday, 21 January 2012

#MONITORINGSUCKS

Saturday, 21 January 2012

SCALABILITY

Saturday, 21 January 2012

execute large volumes of monitoring checks under a variety of conditions

(good + bad) with a consistent throughput

Saturday, 21 January 2012

CONSISTENT THROUGHPUT

Saturday, 21 January 2012

SHORT CHECK EXECUTION TIME

Saturday, 21 January 2012

WHAT INTRODUCES VARIABILITY?

Saturday, 21 January 2012

LATENCY INTRODUCED c SYNCHRONOUS CALLS

Saturday, 21 January 2012

Saturday, 21 January 2012

= 1 monitoring check

Saturday, 21 January 2012

Saturday, 21 January 2012

150 monitoring checkseach executed every 300 secondseach takes 1 secondchecks are executed serially

Saturday, 21 January 2012

all checks executed in 150 secondsmonitoring system at 50% capacity

Saturday, 21 January 2012

DOUBLE THE CHECKS

Saturday, 21 January 2012

Saturday, 21 January 2012

all checks executed in 300 secondsmonitoring system at 100% capacity

Saturday, 21 January 2012

DOUBLE THE EXECUTION TIME

Saturday, 21 January 2012

Saturday, 21 January 2012

all checks executed in 600 secondsmonitoring system at 200% capacityonly 50% of the checks are “on time”

Saturday, 21 January 2012

CHECK LATENCY

Saturday, 21 January 2012

Saturday, 21 January 2012

HOW DO WE FIX THIS!?

Saturday, 21 January 2012

PERFORMANCE ANALYSIS!

Saturday, 21 January 2012

AN ANALOGY

Saturday, 21 January 2012

MONITORING CHECK ==

“ACTION” ON MVC WEB APP

Saturday, 21 January 2012

Saturday, 21 January 2012

Saturday, 21 January 2012

SEPARATE DATA COLLECTION

FROM

THRESHOLDINGf NOTIFICATIONS

Saturday, 21 January 2012

Saturday, 21 January 2012

THIS SHIFTS LATENCY

Saturday, 21 January 2012

IT DOES NOT ELIMINATE IT!

Saturday, 21 January 2012

Saturday, 21 January 2012

RRDTOOL IS EVIL

Saturday, 21 January 2012

USE SOMETHING BETTER!

Saturday, 21 January 2012

USE

OPENTSDBOR

GANGLIAWITH

CHECK_TSDBOR

CHECK_GMOND

Saturday, 21 January 2012

Saturday, 21 January 2012

A DIFFERENT SET a PROBLEMS

Saturday, 21 January 2012

STORAGE WILL GO AWAY

Saturday, 21 January 2012

CHAOS WILL ENSUE

Saturday, 21 January 2012

PAGERS WILL MELT

Saturday, 21 January 2012

SET UP “META-

PARENTING”

Saturday, 21 January 2012

BUILD A KILL SWITCH

Saturday, 21 January 2012

Saturday, 21 January 2012

READ MORE ABOUT THIS:

Saturday, 21 January 2012

bit.ly/yN4mdy

Saturday, 21 January 2012