Who Watches the Watchmen - Arup Chakrabarti, PagerDuty - DevOpsDays Tel Aviv 2015

Who Watches the Watchmen? DevOps Days Chicago 2014

Arup Chakrabarti @arupchak or [email protected]

@arupchak

What is PagerDuty?

10/23/14

Ops Guys know all too well...

•  Alert and Incident Tracking • On-Call Management •  Integrates with monitoring tools •  Alert the right person, every time


@arupchak

What is PagerDuty?

10/23/14 Who Watches the Watchmen? DevOps Days Chicago 2014

@arupchak

Why do we care about monitoring?

10/23/14

Oct 2014 US East Outage – Outgoing Traffic


@arupchak

Today’s talk is about:

10/23/14

•  What is PagerDuty? •  Philosophies •  Tools •  Security •  Distributed Systems •  Dependency •  How we cheat by using Chef •  Validation •  Q and A


@arupchak

Quick Disclaimer

10/23/14

I did not come up with everything


•  I work with smart people •  Slides will be posted

@arupchak

Philosophies

10/23/14

Thou Shall:

•  Use the right tool •  Avoid single host

monitoring


@arupchak

Philosophies

10/23/14

Thou Shall:

•  Alert on what customers care about


@arupchak

Philosophies

10/23/14

Thou Shall:

•  Alert on expected values •  High and Low


@arupchak

Philosophies

10/23/14

Thou Shall:

•  Make it Self-Service


@arupchak

Philosophies

10/23/14

Thou Shall:

•  Validate Alerts Work


@arupchak

Monitoring Tools New Relic


@arupchak

New Relic Oooooo Graphs


@arupchak

New Relic Stacked lines!


@arupchak

New Relic Lines AND Bars!


@arupchak

New Relic Reports!


@arupchak

New Relic Moar Reports!


@arupchak

New Relic / APMs

10/23/14

Great for small env’s

•  Pros •  Great for new stacks •  Helpful for tracing transactions •  Gives a lot of data


@arupchak

Problem with APMs

10/23/14

Not a silver bullet

•  Cons •  They can be overly prescriptive •  They can be hard to tune/

customize •  Gives a lot of data


@arupchak

StatsD / DataDog

10/23/14

All hail self service metrics

•  StatsD is the client •  DataDog is the backend •  Super easy to use •  statsd.gauge(metric_name, val) •  statsd.counter(metric_name) •  statsd.histogram(metric_name,val)


@arupchak

StatsD / DataDog Custom Alerts


@arupchak

StatsD / DataDog

10/23/14

Custom Notifications

•  PagerDuty Integration •  Email •  HipChat


@arupchak

StatsD / DataDog

10/23/14

Customize all the things

•  Pros •  Very customizable •  Can change as you grow •  Self Service


@arupchak

StatsD / DataDog

10/23/14

Needs some hand holding

•  Cons •  Need to have Configuration

Management •  Hard to ramp teams up


@arupchak

SumoLogic

10/23/14

Logging as Monitoring

•  Ship Critical App Logs •  Engineers setup alerts on patterns •  “Too many 500’s in the last 10m”

•  Somewhat self-service •  Initial setup is in Chef

•  Hard to use for realtime debugging


@arupchak

PagerDuty at PagerDuty?


@arupchak

PagerDuty at PagerDuty?


YES

@arupchak

Simple External Monitoring

10/23/14

Dumb health checks

•  Wormly and Monitis •  Simple tools •  Backup alerting •  Very naive in the health checks •  Had to build out smarter health

check page


@arupchak

Simple External Monitoring

10/23/14

Dumb health checks made smarter

•  Health Check Page •  Lightly touches internal services •  Gives back an expected value for

each service •  Alert on non-expected value


@arupchak

Security Monitoring

10/23/14

Why do security monitoring?

•  Audits are tedious •  Continuous Audits

•  Earlier alerting •  Easier fixing


@arupchak

Security Monitoring

10/23/14

Audits are tedious

•  IDS via OSSEC •  Monitor Logs / Checksum Dir’s

•  Port scanning •  nmap

•  Scrape IPSec data


@arupchak

Distributed Systems

10/23/14

The single host does not matter anymore

•  Alert on cluster level metrics •  Overall number of 500’s •  % of nodes down •  Overall latency


@arupchak

Avoid Single Host Alerts

10/23/14

Crons should not be used for creating alerts


PAGERDUTY

US West 1

US West 2

Linode Monitoring System

@arupchak

Same model for service alerting


PAGERDUTY

Service A

Service B

Service C Monitoring System

@arupchak

Dependency Monitoring

10/23/14

Stuff that you do not control

•  Dependencies Everywhere •  Operations •  DNS •  Monitoring Tools •  Logging


@arupchak


10/23/14

Stuff that you do not control

•  How to monitor? •  Operations •  DNS -> Create/Delete records •  Monitoring Tools -> Basic ping •  Logging -> Validate that logs are

being pushed •  Status Pages


@arupchak


10/23/14

What keeps us up at night

•  Dependencies Everywhere •  Software •  Email •  SMS •  Phone •  Push Notifications


@arupchak

Quick Story

10/23/14

When SMS providers screw us over

•  Primary SMS provider was “Up” •  Customer was not getting their SMS •  Found out in the worst way possible •  Customer called us

•  Provider was working but T-Mobile prepaid was not passing our short code through


@arupchak

End to End testing

10/23/14

aka how to abuse unlimited messaging plans

•  Every minute we send a SMS alert •  Every SMS provider we use •  Main Carriers •  Verizon •  AT&T •  T-Mobile •  Sprint

•  Measure Response times


@arupchak

“Device Lab” It looks more official now


@arupchak

Some stats (Averages)

10/23/14

Sorry, cannot tell you which carrier is which

•  Carrier A •  15 Seconds

•  Carrier B •  5 Seconds

•  Carrier C •  15 Seconds

•  Carrier D •  50 Seconds


@arupchak

Very Spikey 10/23/14 Who Watches the Watchmen? DevOps Days Chicago 2014

@arupchak

How to cheat with Chef

10/23/14

Automate all the things

•  Install all the agents •  New Relic •  DataDog – Easy alerts as well •  SumoLogic •  OSSEC

•  Backup alerts are not automated •  Cluster alert setup is not automated


@arupchak

How to Validate

10/23/14

Failure Friday

•  We attack our own services •  Process failure •  Datacenter failure •  Network failure

•  https://blog.pagerduty.com/failure-friday-at-pagerduty


@arupchak

What we have learned

10/23/14

Failure Friday

•  Process monitoring is co-mingled with the process running

•  Only localhost checks on service •  Alerts require outbound network conn


Thank you. We are Hiring! pagerduty.com/jobs

@arupchak or [email protected]

Who Watches the Watchmen - Arup Chakrabarti, PagerDuty - DevOpsDays Tel Aviv 2015

Technology

Transcript of Who Watches the Watchmen - Arup Chakrabarti, PagerDuty - DevOpsDays Tel Aviv 2015