Beyond VM deploymentBeyond VM deploymentMonitoring your VM's at scale

Kris Buytaert

Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source

Consultant @inuits.euConsultant @inuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops● But mostly, trying to be good at my jobBut mostly, trying to be good at my job

What's different in the cloud ?What's different in the cloud ?

● ScaleScale

● VelocityVelocity

● ChangeChange

ChallengesChallenges● ReproducabilityReproducability

● SpeedSpeed

● AuditingAuditing

● Keeping stuff in sync Keeping stuff in sync

• MonitoringMonitoring

• SecuritySecurity

Case :Case :Using a configuration management Using a configuration management tool to configure, update and keep tool to configure, update and keep your cloudscale monitoring and metric your cloudscale monitoring and metric infrastructure sane and manageable.infrastructure sane and manageable.

ToolsTools● PuppetPuppet

● CollectdCollectd

● GraphiteGraphite

● Nagios / IcingaNagios / Icinga

● Chef CfengineChef Cfengine

● GangliaGanglia

● SensuSensu

Not quite a Muppet.Not quite a Muppet.

● Puppet is...Puppet is...

● OSSOSS

● A DSL languageA DSL language

● Written in RubyWritten in Ruby

● Client/server orientedClient/server oriented

● Contains abstraction layersContains abstraction layers

● Repeatable processesRepeatable processes

Master of PuppetsMaster of Puppets● Puppet masterPuppet master

• CA authorityCA authority

• Hosts ModulesHosts Modules

• Hosts Node descriptionsHosts Node descriptions

• Compare, compile, applyCompare, compile, apply

● Master is not a requirement !Master is not a requirement !

Puppet ClientsPuppet Clients● daemondaemon

● Cron jobsCron jobs

● External orchestration:External orchestration:

• for i in $hosts; do ssh $i “puppetd --test”; donefor i in $hosts; do ssh $i “puppetd --test”; done

• mCollective, Func, …mCollective, Func, …

● Get catalogs, play them, Get catalogs, play them,

● reportingreporting

Puppet EnvironmentsPuppet Environments

● Different code bases on 1 masterDifferent code bases on 1 master

● Dev, Uat, ProdDev, Uat, Prod

● Only break one environment at once :)Only break one environment at once :)

● What about testing your Puppetmaster ?What about testing your Puppetmaster ?

Node definitionsNode definitions● Nodes.ppNodes.pp

cclass defaults {lass defaults {$search = "inuits.be"$search = "inuits.be"$nameservers = ['208.67.220.220', '208.67.222.222']$nameservers = ['208.67.220.220', '208.67.222.222']

include dns::resolvinclude dns::resolvinclude ssh::keysinclude ssh::keysinclude ssh::serverinclude ssh::server

}}

node "ns1.dev.inuits.be" {node "ns1.dev.inuits.be" {include defaultsinclude defaultsinclude dns::powerdns::serverinclude dns::powerdns::serverinclude dns::powerdns::resolverinclude dns::powerdns::resolver

}}

node “web1.dev.inuits.be” {node “web1.dev.inuits.be” {include defaultsinclude defaultsinclude apache2include apache2include mysqlinclude mysql

}}

External Node ClassifierExternal Node Classifier● Fixed hostname ? Fixed hostname ?

● How many nodes How many nodes

● Naming schemas solve some issues Naming schemas solve some issues

● External script that sends back yaml class descriptionsExternal script that sends back yaml class descriptions

• Custom writttenCustom writtten

• Foreman Foreman

• ... ...

Classes vs ModulesClasses vs Modules● Module : Module :

● Abstract definition on configuring a serviceAbstract definition on configuring a service

● ReusableReusable

● Class : Class :

● Specific implementation of your use case of Specific implementation of your use case of such a module such a module

•e.g usernames / passwords / hosts do not belong in e.g usernames / passwords / hosts do not belong in modulesmodules

ModulesModules● FilesFiles

● TemplatesTemplates

● ManifestsManifests

• DSLDSL

• ClassesClasses

• ElementsElements

Parametrized Parametrized ClassesClasses

Stored ConfigsStored Configs

Use Cases:Use Cases:● Ssh keysSsh keys

● Reverse proxy configsReverse proxy configs

● Monitoring resourcesMonitoring resources

● Measuring resourcesMeasuring resources

Collection and ExportCollection and Export

Export :Export :

@@resource { @@resource {

... }... }

Collect:Collect:

Resource <<| Resource <<| query |>>query |>>

Clean out nodes that dissapearClean out nodes that dissapear

puppet node clean puppet node clean

Defining a ServiceDefining a Service● Local class that :Local class that :

• Configures service using a standard Configures service using a standard module call with hiera based parametersmodule call with hiera based parameters

• Configures BackupConfigures Backup

• Configures logrotation Configures logrotation

• Configures logshipping Configures logshipping

• Exports Monitoring NeedsExports Monitoring Needs

● Abuse modules for git easeAbuse modules for git ease

Apache Example:Apache Example:

#monitoringsucks#monitoringsucks

Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love: Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love:

● Having my hands tied with the model of host and service bindings. Having my hands tied with the model of host and service bindings.

● Having to set up "fake" hosts just to group arbitrary metrics together Having to set up "fake" hosts just to group arbitrary metrics together

● Having to either collect metrics twice - once for alerting and another for trending Having to either collect metrics twice - once for alerting and another for trending

● Only being able to see my metrics in 5 minute intervals Only being able to see my metrics in 5 minute intervals

● Having to chose between shitty interface but great monitoring or shitty monitoring but Having to chose between shitty interface but great monitoring or shitty monitoring but great interface great interface

● Dealing with a monitoring system that thinks IT is the system of truth for my Dealing with a monitoring system that thinks IT is the system of truth for my environment environment

● Not actually having any real choicesNot actually having any real choices

John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-monitoring-sucks.htmlmonitoring-sucks.html

#monitoringlove#monitoringlove● Puppet Puppet

● Nagios (Icinga)Nagios (Icinga)

● GraphiteGraphite

● CollectdCollectd

● LogstashLogstash

Graphite Graphite ● Graphing at ScaleGraphing at Scale

● Graphing at EaseGraphing at Ease

● Any metric is a graphAny metric is a graph

● echo "somestring $somevalue echo "somestring $somevalue $timestamp" | nc <%= graphitehost $timestamp" | nc <%= graphitehost %> 2003%> 2003

Graphite ComposerGraphite Composer

x

Graphite APIGraphite API

Gdash In actionGdash In action

Puppet and GraphitePuppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/https://github.com/KrisBuytaert/vagrant-graphite/

● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules ! Statsd / Tattle and more modules as submodules !

● git clonegit clone

● git submodule init git submodule init

● git submodule update git submodule update

● vagrant up vagrant up

CollectdCollectd● CollectsCollects

● Zillion PluginsZillion Plugins

• Nginx,apache, mysql, diskNginx,apache, mysql, disk

● Graphite Carbon PluginGraphite Carbon Plugin

● Send metrics to graphiteSend metrics to graphite

Collectd & GraphiteCollectd & Graphite

Exporting and Collecting Exporting and Collecting

Triggers on GraphsTriggers on Graphs● Export Java MetricsExport Java Metrics

● JMXTransJMXTrans

● Export JMXConfigsExport JMXConfigs

● Configure NRPE Configure NRPE CheckCheck

● Export NagiosCheckExport NagiosCheck

● Collect JMX Exports Collect JMX Exports on JMXTransNodeon JMXTransNode

● Graph EmGraph Em

Collect Nagios Collect Nagios Configs on Nagios Configs on Nagios ServerServer

Triggers on GraphsTriggers on Graphs

Triggers on GraphsTriggers on Graphs

Conclusion:Conclusion:● Reproducable monitoring setupReproducable monitoring setup

● Dynamically generated monitoring Dynamically generated monitoring configconfig

● Code is available at Code is available at github.com/KrisBuytaert/snippetsgithub.com/KrisBuytaert/snippets

ContactContactKris Buytaert Kris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be

Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.be/http://www.inuits.be/

InuitsInuits

Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231

+32 475 961221+32 475 961221