Sock Puppets – Phase One An Investigation into Building Effective ...
Sock Puppets: Growing Your Puppet Codebase
-
Upload
puppet-labs -
Category
Technology
-
view
1.995 -
download
3
description
Transcript of Sock Puppets: Growing Your Puppet Codebase
Sock Puppets: Growing your puppet codebase
Fabio Lessa & Andrew Myers
Part 1
Empty server room
Project teams
teams
OPS team for app support
Something that represents a mess
The codebase we've started with
Standard Puppet Master setup
Code was split by environments
Workflow: 1. Push change 2. Wait for it to propagate to that environment 3. Check it worked If you need that applied to another environment, go to 1
The real Workflow: 1. Push change 1.1. Stop the puppet daemon in all the critical nodes for that environment 1.2. Run puppetd manually in one host to ensure it works 1.3. Start the daemon again (on all the hosts you remember) 2. Wait for it to propagate to that environment 3. Check it worked (meh) If you need that applied to another environment, go to 1 (if you remember, that is)
UAT
Production
System Testing
CI
Making changes was painful
Part 2
"don't duplicate your manifests in your tests" Focus on the catalog problems he described earlier and test your logic. Don't test if puppet is doing it's job,
test that your logic it's doing it's job.
- Nikolay Sturm
Before After
The goal...
So how do we test/refactor
Compile Puppet Catalogues
Compare Versions
Application specific tests
Tying it all together
Part 3
Continuous Delivery
We were moving application deploys, why not do the same with infrastructure?
MCollective
The new setup
Picture of the pipelines
info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/hostinfo/lib/facter/hostinfo.rb info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/concat/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: Could not find dependency File[/root/.hgrc] for Mercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/puppetmaster/manifests/repos.pp:36
info: Retrieving plugin info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: You cannot specify more than one of content, source, target at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/sbs/manifests/init.pp:73
notice: Class[Hostinfo]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message: current_value absent, should be Setting java the following java as default: 1.6.0.31 sun 1.6.0 (noop) notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered 'refresh' from 1 events notice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggered 'refresh' from 1 events notice: Class[Java::Sun]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_value absent, should be directory (noop) err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could not evaluate: Could not retrieve information from environment noop source(s) puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/manifests/init.pp:30 notice: Class[Puppetagent]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_value directory, should be absent (noop) notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure: current_value file, should be absent (noop)
We were moving application deploys, why not do the same with infrastructure?
Our MCollective agent
The new action
At every push, all nodes connect to the noop environment and run in noop mode
Picture of a report
Change management process ?
What changed?
What did you test?
Who will do it?
Part 4
New data centre is ready!
Should be easy to migrate....
Okay so our manifests are incomplete, but we need to be able to rebuild machines...
Virtualisation to the rescue...
Magnets?
What happens
Wrap Up
120 pushes to production in the last 8 months: 2 failures
5 minutes!
MTTR FTW!
Thank you!