Continuous Delivery of your Infrastructure

Kris Buytaert@krisbuytaert

Kris Buytaert

I used to be a Dev,

Then Became an Op

Chief Trolling Officer and Open Source Consultant @inuits.eu

Everything is an effing DNS Problem

Building Clouds since before the bookstore

Some books, some papers, some blogs

Evangelizing devops

Todays Goals

A reproducable way to deploy and upgrade your infrastructure

Automatically

Fast

Consistent

Continuously

What's this devops thing anyhow ?

C(L)AMS

Culture

(Lean)

Automation

Measurement

SharingDamon Edwards and John Willis

Gene Kim

Whats in it for you ?

Faster time to marketFeatures go live in hours vs years

In a more safe (Secure)

Reliable fashionFully automated

More happy {customers,developers,ops,managers,investors}

Less Risk

devops ( continuous delilvery

Nirvana

An ecosystem that supports continuous delivery, from infrastructure, data and configuration management to business.Through automation of the build, deployment, and testing process, and improved collaboration between developers, testers, and operations, delivery teams can get changes released in a matter of hours sometimes even minutesno matter what the size of a project or the complexity of its code base.Continuous Delivery , Jez Humble

How many times a day ?

10 @ Flickr

Deployments used to be pain

Nobody dared to deploy a site

Practice makes perfect

Knowing you can vs constantly doing it

" Our job as engineers (and ops, dev-ops, QA, support, everyone in the company actually) is to enable the business goals. We strongly feel that in order to do that you must have the ability to deploy code quickly and safely. Even if the business goals are to deploy strongly QAd code once a month at 3am (its not for us, we push all the time), having a reliable and easy deployment should be non-negotiable." Etsy Blog upon releasing Deployinatorhttp://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/

For years we've tolerated humans to make structural manual changes to the infrastructure our critical applications are running on.Whilst at the same time demanding those critical applications to go through rigid test scenarios.Who let this happen ?

Culture Hack:

Set up CI / CD for your infrastructure first, If the people running your infra don't know how CI/CD works , how do you expect them to support / teach your application teams ?You also get them to learn about the tooling they will need to support and they will share the pain and the joy of the application developers

Why start with infra ?

Learn

Be able to support your peers

It's a test requirement anyhow

Automate all the things

Buildreproducable builds are undiscussable

Testtesting reduces risk

automate deployments of your test infra

DeployInfrastructure as Code

100% automation

Can you rebuild your infrastructure ?

Use Version Control,

No Excuses

Also for scripts/config/cookbooks,manifests,etc

Infrastructure as Code

Treat configuration automation as code

Development best practicesModel your infrastructure

Version your cookbooks / manifests

Test your cookbooks/ manifests

Dev/ test /uat / prod for your infra

Model your infrastructure

A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )

IAC -ne scripting

Release Management

What parts go in the build ?

What are the dependencies ?

What versions work together ?

Every frameworks invents it's own.None work

Release Management

Git Submodules !

Submodules and you will never need a Release Management tool again , ever

Tool independent pattern, Puppet

Drupal

Symfony

.

CI Tools

Hudson

Jenkins

A zillion plugins

Go (Cruise Control)?

Travis ?

Also test your (Puppet/Chef/CFengine)

Build Pipelines

Jenkins

Open Source Continuous Integration Server

A zillion plugins (400)

Have developers build stable and deployable code

Test Infra code

Jenkins Pipeline

It gets harder

Try to fully automate JenkinsIncluding plugin config

Including all the jobs

Try templating the jobs JJB

Job DSL

What's in your Pipeline ?

A pipeline

Checkout code

Syntax

Style

Code Coverage

Tests

Build

More Tests

Package

Syntax and Style

Initially , all code, all the time

Now, only the changed code

Why not in post Commit Hooks ?

Package all the things

Why ops like to package

Packages give you features

Consistency, security, dependencies

Uniquely identify where files come from

Package or cfg-mgmt

Source repo not always available

Firewall / Cloud etc ..

Weird deployment locations , no easy access

Little overhead when you automate

#packaginlove

Say thanks to Jordan Sissel

It's not really packaging

It's an immutable branch

It's a tracable release artefact

A pipeline

Checkout code

Syntax

Style

Code Coverage

Tests

Build

More Tests

Package

Upload to Repo

Repository Management

PulpPro : MirroringLove

Con : Mongo, Stability, .deb

PRM ?

https://github.com/ImmobilienScout24/yum-repo-server ?

Repository Management

Deploy Software

Repo per environment

Package {'mysoftware': ensure => latest}You control the repos !

Strict versioning in config ?Ensure => '0.98.4'

A pipeline

Checkout code

Syntax

Style

Code Coverage

Tests

Build

More Tests

Package

Upload to Repo

Deploy on Test

Repos are SLOW

Createrepo is slow.

Pulp is slow

Bypass repos , upload straight to appropriate PuppetMaster

Upload to repo for rebootstrapping

Orchestration

Manage 1000 nodes,

Trigger Upgrades

Config Runs

Service Changes

Think : Mcollective

Consul

Ansible

Rundeck

Orchestration 2nd gen

Aka ChoreographyWhile ....

First install X

When it is ready configure Y

Then notify Z

Think : Zookeeper, Serf , Juju

REARCHITECT !

Testing your infra

One Icinga check at the time

Roll out new (puppet) code

Trigger puppet runs on environmentLoad on Puppetmaster , but more on Ansible

Trigger check icinga

Reuse tests for monitoring

Unit tests vs Acceptance Tests

Your monitoring platform also has a test / uat / prod flow

Is fully automated

Keeping UAT green is a must This is where you learn PROD behaviour

A pipeline

Checkout code

Syntax

Style

Code Coverage

Tests

Build

More Tests

Package

Upload to Repo

Deploy on Test

Check Puppetruns

Check Icinga

Promote to UAT

Jenkins Promotion

A pipeline

Checkout code

Syntax

Style

Code Coverage

Tests

Build

More Tests

Package

Upload to Repo

Deploy on Test

Check Puppetruns

Check Icinga

Promote to UAT

Test even more , security etc..

Promote to PROD

Attack yourselve on every build

Gauntlt , write security tests

Vulnerability scans (Arachni)

Content Scanner (DIRB)

https://github.com/garethr/pentesting-playground

Done ?

Close the feedback loop,

Send metric on deployment

echo "deployed.$package_name 1 `date +%s`" > /dev/tcp//2003

No more

Hacking in production

Can't change this

I don't have monitoring for this

Contact

Kris Buytaert Kris.Buytaert@inuits.be

Further Reading@krisbuytaert http://www.krisbuytaert.be/blog/http://www.inuits.be/

Inuits

Essensteenweg 31BrasschaatBelgium891.514.231

+32 475 961221