Continuously deliver your puppet code withjenkins, r10k and git

Toni Schmidbauer

October 14, 2014

whoami

I SysAdmin@s-itsolutions.at

I toni@stderr.at

I stderr@jabber.org

I http://stderr.at

I http://github.com/tosmi

Agenda

I A short story about configuration management

I What is continuous delivery

I Tools used to achieve continuous delivery

I DEMO

I Things to improve

A short story about configuration management (CM)

I We manage a very diverse environment of UNIX/LinuxSystems (Solaris 10/11 SPARC/i386, AIX, RHEL/CentOS5/6/7)

I We’ve got around 1000 nodes in total

I Before CM we had strict standards on how to manage thesesystems

I The problem:count(team members) == count(standards)

I So configuration management is the solution to all ourproblems

A short story about configuration management (CM)

I We manage a very diverse environment of UNIX/LinuxSystems (Solaris 10/11 SPARC/i386, AIX, RHEL/CentOS5/6/7)

I We’ve got around 1000 nodes in total

I Before CM we had strict standards on how to manage thesesystems

I The problem:count(team members) == count(standards)

I So configuration management is the solution to all ourproblems

A short story about configuration management (CM)

I We manage a very diverse environment of UNIX/LinuxSystems (Solaris 10/11 SPARC/i386, AIX, RHEL/CentOS5/6/7)

I We’ve got around 1000 nodes in total

I Before CM we had strict standards on how to manage thesesystems

I The problem:count(team members) == count(standards)

I So configuration management is the solution to all ourproblems

A short story about configuration management (CM)

I We manage a very diverse environment of UNIX/LinuxSystems (Solaris 10/11 SPARC/i386, AIX, RHEL/CentOS5/6/7)

I We’ve got around 1000 nodes in total

I Before CM we had strict standards on how to manage thesesystems

I The problem:count(team members) == count(standards)

I So configuration management is the solution to all ourproblems

A short story about configuration management (CM)

I We manage a very diverse environment of UNIX/LinuxSystems (Solaris 10/11 SPARC/i386, AIX, RHEL/CentOS5/6/7)

I We’ve got around 1000 nodes in total

I Before CM we had strict standards on how to manage thesesystems

I The problem:count(team members) == count(standards)

I So configuration management is the solution to all ourproblems

The solution to all our problems

I Broke our systems

The solution to all our problems

I Broke our systems

WHY????

Problems with our old CM system

I Deployments sucked

I Deployment via manual tagging and checkout, so mistakeshappened

I Deployment in stages, but we always had to cross our fingers

I Testing sucked

I No UnittestsI No acceptance tests

I No immediate feedback if things where OK or not

I Systems installed without CM are hard to bring under CMcontrol

I Every system was a special case

Problems with our old CM system

I Deployments suckedI Deployment via manual tagging and checkout, so mistakes

happenedI Deployment in stages, but we always had to cross our fingers

I Testing sucked

I No UnittestsI No acceptance tests

I No immediate feedback if things where OK or not

I Systems installed without CM are hard to bring under CMcontrol

I Every system was a special case

Problems with our old CM system

I Deployments suckedI Deployment via manual tagging and checkout, so mistakes

happenedI Deployment in stages, but we always had to cross our fingers

I Testing sucked

I No UnittestsI No acceptance tests

I No immediate feedback if things where OK or not

I Systems installed without CM are hard to bring under CMcontrol

I Every system was a special case

Problems with our old CM system

I Deployments suckedI Deployment via manual tagging and checkout, so mistakes

happenedI Deployment in stages, but we always had to cross our fingers

I Testing suckedI No UnittestsI No acceptance tests

I No immediate feedback if things where OK or not

I Systems installed without CM are hard to bring under CMcontrol

I Every system was a special case

Problems with our old CM system

I Deployments suckedI Deployment via manual tagging and checkout, so mistakes

happenedI Deployment in stages, but we always had to cross our fingers

I Testing suckedI No UnittestsI No acceptance tests

I No immediate feedback if things where OK or not

I Systems installed without CM are hard to bring under CMcontrol

I Every system was a special case

Problems with our old CM system

I Deployments suckedI Deployment via manual tagging and checkout, so mistakes

happenedI Deployment in stages, but we always had to cross our fingers

I Testing suckedI No UnittestsI No acceptance tests

I No immediate feedback if things where OK or not

I Systems installed without CM are hard to bring under CMcontrol

I Every system was a special case

Problems with our old CM system

I Deployments suckedI Deployment via manual tagging and checkout, so mistakes

happenedI Deployment in stages, but we always had to cross our fingers

I Testing suckedI No UnittestsI No acceptance tests

I No immediate feedback if things where OK or not

I Systems installed without CM are hard to bring under CMcontrol

I Every system was a special case

So what’s our solution?

Continuous delivery

I is a pattern for getting software from development to release

I this pattern is called the deployment pipeline

1

1Continuous Delivery: Jez Humble, David Farley

Continuous delivery

I is a pattern for getting software from development to release

I this pattern is called the deployment pipeline

1

1Continuous Delivery: Jez Humble, David Farley

The deployment pipeline

Commit StageAutomated unittesting

Automated acceptancetesting

Tag release Deployment

Why are we using continuous delivery

I When you automate your deployment,

I less mistakes will happen and the same mistake will nothappen twice

I less mistakes means less stress when deployingI less stress means you are going to deploy more oftenI more deployments means a more flexible environment

Why are we using continuous delivery

I When you automate your deployment,I less mistakes will happen and the same mistake will not

happen twice

I less mistakes means less stress when deployingI less stress means you are going to deploy more oftenI more deployments means a more flexible environment

Why are we using continuous delivery

I When you automate your deployment,I less mistakes will happen and the same mistake will not

happen twiceI less mistakes means less stress when deploying

I less stress means you are going to deploy more oftenI more deployments means a more flexible environment

Why are we using continuous delivery

I When you automate your deployment,I less mistakes will happen and the same mistake will not

happen twiceI less mistakes means less stress when deployingI less stress means you are going to deploy more often

I more deployments means a more flexible environment

Why are we using continuous delivery

I When you automate your deployment,I less mistakes will happen and the same mistake will not

happen twiceI less mistakes means less stress when deployingI less stress means you are going to deploy more oftenI more deployments means a more flexible environment

Tools to build a deployment pipeline

Jenkins

I Jenkins is an Open Source continuous integration server

I It’s purpose is to execute and monitor jobs

I Jobs are shell scripts or any other thing that’s executable andreturns 0 on success

I Many plugins available to extend Jenkins (e.g. git,build-pipeline, monitor)

I You can link jobs together, thats our pipeline

Jenkins II

Monitoring with Jenkins

GIT

I git push triggers the deployment pipeline

I one central repository for internal modules

I gitolite for access controlI 3 main branches

I developmentI testingI production

I feature branches for new site local modules

I hiera data is in the same repository

GIT

I git push triggers the deployment pipeline

I one central repository for internal modules

I gitolite for access controlI 3 main branches

I developmentI testingI production

I feature branches for new site local modules

I hiera data is in the same repository

GIT repository layout

I modules/:I where r10k stores external (forge, github) modules

I site/:I site local modules, we do not want to share

I hiera/:I our hiera yaml files

I Puppetfile:I config file for r10k that specifies which external modules we

need

I Vagrantfile:I To boot a development puppet environment on your local

workstation

GIT workflow

Puppet developer

Development

Puppet Master

Testing

Puppet Master

Production

Puppet Master

Feature

Branch

Testing

Branch

Production

Branch

4

3

2

1

Production Nodes

Testing Nodes (~40)

Dev Nodes (7−8)

1

3

2

4

1 Features Branches get automatically created on Puppet Master (Dynamic Environments)

2 Development Branch gets deployed on commit via Jenkins

3 Testing Branch gets deployed via GIT tag

4

GIT

Jenkins

It’s all the same for Hiera yaml files!

Development

Branch

pushing to testing triggers a deployment

pushing to production triggers a deployment

Production Branch gets deployed via GIT tag

r10k

I a tool to deploy puppet environments and modules

I every git branch gets deployed to a corresponding puppetenvironment

I it also downloads and installs modules from puppetforge orgithub

I in the current version (1.3.2) dependencies have to bemanaged manually

Example Puppetfile

1 f o r g e ’ f o r g e . puppe t l ab s . com ’

3 mod ’ puppe t l ab s /ntp ’ , ’ 3 . 1 . 2 ’mod ’ puppe t l ab s / p o s t g r e s q l ’ , ’ 3 . 4 . 2 ’

5 mod ’ puppe t l ab s / s t d l i b ’ , ’ 4 . 3 . 2 ’mod ’ puppe t l ab s / f i r e w a l l ’ , ’ 1 . 1 . 3 ’

7 mod ’ puppe t l ab s /apache ’ , ’ 1 . 1 . 1 ’mod ’ puppe t l ab s / lvm ’ , ’ 0 . 3 . 2 ’

9 mod ’ n o s o l u t i o n s /tsm ’ , ’ 0 . 2 . 2 ’mod ’ saz / sudo ’ , ’ 3 . 0 . 6 ’

11 mod ’ s p i e t t e / s e l i n u x ’ , ’ 0 . 5 . 4 ’

13 mod ’ concat ’ ,: g i t => ’ h t t p s : // g i t hub . com/ puppe t l ab s / puppe t l abs−concat ’ ,

15 : commit => ’ feba3096c99502219043b8161bde299ba65e7b8a ’

You are able to pin to a git tag / branch / commit hash

a word on testing

I you must have unit tests for your puppet code: rspec-puppet

I for acceptance tests there’s puppetlabs/beaker

I you need to test everything to get most out of the buildpipeline

I we testI internal puppet modulesI hiera dataI puppet configuration

I all internal modules are required to have rspec tests

rspec-puppet example

samplemodule/manifests/init.pp

1 c l a s s samplemodule ( $message = ’ de f au l tmes sage ’ ) {n o t i f y { ’ samplemessage ’ :

3 message => ”This i s the sample module , my message i s : $message ” ,}

5 }

samplemodule/spec/classes/samplemodules spec.rb

1 r e q u i r e ’ s p e c h e l p e r ’

3 d e s c r i b e ’ samplemodule ’ , : t ype => : c l a s s docon t e x t ’ w i th d e f a u l t paramete r s ’ do

5 i t { shou l d c o n t a i n n o t i f y ( ’ samplemessage ’ ) }end

7 end

beaker example

1 r e q u i r e ’ s p e c h e l p e r a c c e p t a n c e ’

3 d e s c r i b e ’ p r o f i l e s : : o s s ba s e c l a s s ’ doi t ’ s hou l d work wi th no e r r o r s ’ do

5 a p p l y man i f e s t ( ’ i n c l u d e p r o f i l e s : : o s sbase ’ , : c a t c h f a i l u r e s => t r u e )end

7 end

DEMO

Do try this at home

I You need:I Vagrant from http://vagrantup.comI VirtualboxI Git client

I You have to run:I git clone

https://github.com/tosmi/puppetcamp2014.gitI cd puppetcamp2014I vagrant upI vagrant ssh

Things we have to improve

I We need more test Systems (CentOS/RHEL/Solaris/AIX?)

I We need more acceptance tests

I Once again use stages in production

I Our documentation

Things puppetlabs should improve

I Puppetlabs should package beaker as a rpm/deb whatever,gems suck in production

Summary

I Continuous Delivery is implemented via a deployment pipeline

I Within the pipeline everything is automated

I When everything is automated you need to test everything

I When everything is automated and well tested deployingbecomes easy

I Tools we are using to implement a deployment pipelineI Jenkins: to execute jobsI GIT: version controlI Gitolite: access control and authorization for GITI r10k: install puppet modules / create puppet environments

from branchesI rspec-puppet: unittests for puppetI puppetlabs/beaker: acceptance tests for puppet

Summary

I Continuous Delivery is implemented via a deployment pipeline

I Within the pipeline everything is automated

I When everything is automated you need to test everything

I When everything is automated and well tested deployingbecomes easy

I Tools we are using to implement a deployment pipelineI Jenkins: to execute jobsI GIT: version controlI Gitolite: access control and authorization for GITI r10k: install puppet modules / create puppet environments

from branchesI rspec-puppet: unittests for puppetI puppetlabs/beaker: acceptance tests for puppet

Summary

I Continuous Delivery is implemented via a deployment pipeline

I Within the pipeline everything is automated

I When everything is automated you need to test everything

I When everything is automated and well tested deployingbecomes easy

I Tools we are using to implement a deployment pipelineI Jenkins: to execute jobsI GIT: version controlI Gitolite: access control and authorization for GITI r10k: install puppet modules / create puppet environments

from branchesI rspec-puppet: unittests for puppetI puppetlabs/beaker: acceptance tests for puppet

Summary

I Continuous Delivery is implemented via a deployment pipeline

I Within the pipeline everything is automated

I When everything is automated you need to test everything

I When everything is automated and well tested deployingbecomes easy

I Tools we are using to implement a deployment pipelineI Jenkins: to execute jobsI GIT: version controlI Gitolite: access control and authorization for GITI r10k: install puppet modules / create puppet environments

from branchesI rspec-puppet: unittests for puppetI puppetlabs/beaker: acceptance tests for puppet

Summary

I Continuous Delivery is implemented via a deployment pipeline

I Within the pipeline everything is automated

I When everything is automated you need to test everything

I When everything is automated and well tested deployingbecomes easy

I Tools we are using to implement a deployment pipelineI Jenkins: to execute jobsI GIT: version controlI Gitolite: access control and authorization for GITI r10k: install puppet modules / create puppet environments

from branchesI rspec-puppet: unittests for puppetI puppetlabs/beaker: acceptance tests for puppet

Thanks for your attention!