Prepared by

Build a modern infrastructure in 45 min!

Matthew Barr Sr. Systems Engineer

Is your infrastructure a

mess?

Let’s fix it :)

What we’re going to do:

• Define a modern infrastructure

• Glance at their architectures

• Demonstrate how to do this yourselves

• … And then the details..

What is a modern infrastructure?

It includes:

• Centralized logging

• Monitoring

• Orchestration

• CI (continuous integration)

• Metrics*

What we’ll do today: Setup

• Mcollective

• Sensu (ideal for cloud infra)

• Logstash + ElasticSearch + Kibana

• Jenkins

MCollective (mco)

• Orchestration

• Uses ActiveMQ or RabbitMQ

• Maintained by Puppet Labs

• http://puppetlabs.com/mcollective

• Distributed monitoring system

• Uses RabbitMQ

• has a easy API

• Adding/remove servers without restarting or changing config files on server

• http://sensuapp.org

Sensu!

Logstash

http://logstash.net/docs/1.4.1/tutorials/getting-started-with-logstash

Elastic Search & Kibana

• Elasticsearch (http://www.elasticsearch.com) is a “distributed restful search and analytics tool”

• It’s used as a datastore for Logstash. (it’s not the only one, but one of the most used.)

• Kibana is a dashboard for use with Elasticsearch & Logstash.

What we’re actually doing:

• Show how to use a set of forge modules to build an infrastructure out.

• using the mbarr/moderninfra as an opinionated profile module

• download the necessary modules using librarian-puppet

We’ll:

• Build a RabbitMQ server + sensu server

• the admin host (has the mco client)

• Build a logstash server

• Build a Jenkins host

Each server will also:

• be sending logs via logstash-forwarder

• run Sensu client checks

• run a mco server

Moderninfra module

A forge module just for you!

• Sets up the basics of each service

• Sets up the requirements correctly to all work together

• Has… opinions.

Install from the forge:

puppet module install mbarr-moderninfra

The code!

---!moderninfra::rmqserver: 'rabbitmq.aws.mbarr.net'!moderninfra::mco_password: 'shhhh..its.a.secret.'!moderninfra::sensu_password: 'whatsupdoc'!moderninfra::logstash_server: 'logstash.aws.mbarr.net'

Hiera data, to make life easier:

class moderninfra (!$rmqserver,!$logstash_server,!$rmq=false,!$mco_client=false,!$mco_server=false,!$sensu_client=false,!$sensu_server=false,!$logstash=false,!$logstash_forwarder=true,!$mco_password=undef,!$sensu_password=undef,!) {...}

node default {! if $role == "mco" {! class {'moderninfra':! rmq => true,! mco_client => true,! sensu_server => true,! }! include profiles::sensuchecks ! }!! if $role == "puppet" {! class {'moderninfra':! mco_server => true,! sensu_client => true,! }! }

if $role == "logstash" {! class {'moderninfra':! logstash => true,! mco_server => true,! sensu_client => true,! }! include profiles::logstash! }!! if $role == "jenkins" {! class {'moderninfra':! mco_server => true,! sensu_client => true,! }! include jenkins! }!}

Site.pp

RabbitMQ, Sensu & Mcollective

RabbitMQ

• This is the middle ware that is used by both mco & sensu.

• Our module uses the Puppet SSL certs for connections

• Adds a second cert for the host, via the puppet-certificate module.

Code

class {'moderninfra':! rmq => true,! mco_client => true,! sensu_server => true,! }! include profiles::sensuchecks ! }

RMQ Note

• To be fair: Sensu isn’t running w/ SSL certs

• I’ve used other self signed certs before without issue

• Looks like there’s a bug that hopefully is actually fixed in Erlang OTP 17.1

Mcollective

• Using SSL to secure PSK connections between mco & RabbitMQ

• Installs the package, service & puppet agents.

root@rmq-us-east-1b-i-6a9bda41:~# mco package status puppet ! * [ ============================================================> ] 4 / 4 ! puppet-us-east-1b-i-346b2a1f.ec2.mbarr.net: puppet-purged. rmq-us-east-1b-i-6a9bda41.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1. logstash-us-east-1b-i-979adbbc.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1. jenkins-us-east-1b-i-969adbbd.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1. !Summary of Arch: ! No aggregate summary could be computed !Summary of Ensure: ! 3.6.2-1puppetlabs1 = 3 purged = 1 !!Finished processing 4 / 4 hosts in 1172.09 ms

Sensu

• Client on all 4 hosts

• Server on RMQ box

• Distributed checks

• Dashboard on 8080

• profiles::sensuchecks installs various checks. (not in module)

Actually making sensu GO: (on server)

class profiles::sensuchecks {! sensu::check { 'check_ntp':! command => 'PATH=$PATH:/usr/lib/nagios/plugins check_ntp_time -H pool.ntp.org -w 20 -c 40',! handlers => 'default',! subscribers => 'general',! standalone => false,! custom => { occurrences => 2 },! }! sensu::check { 'check_cron':! command => '/etc/sensu/plugins/check-procs.rb -p cron -C 1 -c 10 -w 10 ',! handlers => 'default',! subscribers => 'general',! interval => 60,! standalone => false,! custom => { occurrences => 2 },! }!}!

Logstash

• Centralized logging system

• Inputs, Outputs, Filters

• Inputs: syslog, files, redis..

• Outputs:elasticsearch, etc

• Filters: Grok, many others

Logstash profile

class profiles::logstash {!! logstash::configfile { 'basic_config':! source => 'puppet:///modules/profiles/logstash/basic_config',! order => 10! }!! include kibana3!!}!

Logstash configinput { lumberjack { port => 12345 ssl_certificate => "/etc/logstash/ssl/cert.pem" ssl_key => "/etc/logstash/ssl/key.pem" type => "lumberjack" } } !input { tcp { port => 5000 type => syslog } udp { port => 5000 type => syslog } } !output { elasticsearch { host => localhost } stdout { codec => rubydebug } }

Logstash-forwarder

• Data is sent from logs on client to Logstash server via SSL

• Keeps track of log positions and what’s been sent

• Server listens on 12345, for now.

Elasticsearch & Kibana

• This is what Kibana looks like with data from logstash fed into elasticsearch

• (It’s zoomed a bit, so you can see the good parts.)

Jenkins

Jenkins

• Continuous integration tool

• There is code to set up slaves in the Jenkins module.

• https://forge.puppetlabs.com/rtyler/jenkins

include jenkins

Things this module doesn’t do:

• Build your puppet master

• DNS names for Puppet master, RMQ, Logstash, etc

• Although the cloud formation templates do!

But it might let you sleep at

night…

Appendix:!Puppet Master

• Built w/ CloudFormations template

• Sorry, not vagrant. Might be added soon.

• uses cloud-init to provision puppet & code base

• Uses puppet 3.6.2

• Librarian-puppet

Puppet Master

• Set host name & domain

• Install puppet

• rm -rf /etc/puppet

• git clone REPO /etc/puppet

Appendix: !Librarian-puppet

Librarian Puppet

• Lets you take a Puppetfile, and manage modules & dependencies

• can use forge or git repos

• Takes over your modules directory, though.

• adds to .gitignore & regenerates the directory from the Puppetfile

• I’ve used a pattern of a second directory (modules-local) to allow a slow migration & local files to stay in your existing repo

Modules-local pattern

Old: modulepath = $confdir/modules:$confdir/modules-local !3.6+ directory environments: environment.conf modulepath = modules:modules-local

Puppetfile

forge "https://forgeapi.puppetlabs.com" !mod "reidmv/puppet_certificate" mod "elasticsearch/logstash" mod "elasticsearch/elasticsearch" mod "sensu/sensu" !mod "rtyler/jenkins" !mod "puppetlabs/mcollective" !mod "thejandroman/kibana3", "0.0.3" !# mod "mbarr/moderninfra", # :git => "git://github.com/matthewbarr/moderninfra.git" !#mod "garethr/graphite"

modules ├── activemq ├── apache ├── apt ├── concat ├── datacat ├── elasticsearch ├── epel ├── erlang ├── file_concat ├── git ├── java ├── java_ks ├── jenkins ├── kibana3 ├── logstash ├── mcollective ├── puppet_certificate ├── rabbitmq ├── sensu ├── staging ├── stdlib ├── vcsrepo └── zypprepo modules-local ├── moderninfra └── profiles

We’re hiring! (in Boston)!!

!

Matthew Barr!@matthewbarr (github & twitter)!matthew.barr@here.com!mbarr@mbarr.net!

http://github.com/matthewbarr/build-modern-infra