Enjoying the Journey from Puppet 3.x to Puppet 4.x Rob Nelson

Who Am I?

Puppet user since 2014 (3.3 era) Vox Pupuli, puppet-lint contributor @rnelson0, https://rnelson0.com/

Agenda •  Why upgrade?

•  Refactor our codebase for Puppet 4

•  Upgrade our Puppet master(s) and agents to Puppet 4.x

•  Refactor our codebase to remove Puppet 2- and 3-isms

•  Tips, tricks, and tools

•  Enjoying ourselves

Why? •  Puppet 4 is old! First released March, 2015

•  Puppet 3 is really old! End Of Support on December 31, 2016

•  Puppet 4 only modules

•  Puppet 4 language improvements

•  Application Orchestration

•  PE first, FOSS eventually; some free implementations (such as choria) appearing

•  AIO Puppet and Puppetserver

•  Better performance, security; same agent/puppetserver between FOSS and PE

•  Puppet 5 is coming!

Who does this apply to? •  Puppet Enterprise users

•  Puppet Opensource users

•  Foreman (1.13+) users

•  Master and Masterless

Blueprint •  Start with Puppet 3.x

•  Read the release notes

•  Plan the roadmap

•  Validate / create tests

•  Refactor until your current version passes all tests

•  Upgrade/Replace the master(s)

•  Upgrade the agents

•  Repeat the Refactor / Upgrade steps until you get to 4.latest

Release Notes •  All of them - not just the latest version

•  Identify potential issues, deprecated features, etc

•  Determine the minimum version required to upgrade to target version

•  Stay up to date

Define the Roadmap •  Determine the current version

•  In-place upgrades or new infrastructure?

•  Identify intermediate version steps

•  Enable Future Parser [and Strict Variables] before you hit 4.x

•  PE: Requires intermediate upgrades or fresh installs (check KB)

•  FOSS: Go straight to 4.latest

•  Determine how upgrades and interruptions affect ecosystem products – PE Console/puppetboard, SEIMs, monitoring, etc.

FOSS Example Roadmap •  3.6.0 -> 3.8.7

•  3.8.7 w/Future Parser [and Strict Variables]

•  3.8.7 -> 4.7.0

•  Today’s example roadmap

PE Example Roadmap •  3.7.2 -> 3.8.6

•  3.8.6 w/Future Parser [and Strict Variables]

•  3.8.6 -> 2015.3.3

•  2015.3.3 -> 2016.2.1

Validate/Create Tests •  Tests assure (mostly) predictable behavior

•  Determine what kinds of tests you need - unit, acceptance, integration, other?

•  Good testing setup in puppet-module-skeleton

•  Never written an rspec-puppet test? puppet-retrospec generates naive tests that need tuned

•  Existing tests must pass before modifying code

•  Turn on Future Parser [and Strict Variables] only at 3.8.x

•  Beyond tests: catalog diffs, personalized tests

Rspec Tests $ cat spec/classes/apache_spec.rb require 'spec_helper' describe 'profile::apache', :type => :class do let :facts do { facts_hash } end context 'with defaults for all parameters' do it { is_expected.to create_class('profile::apache') } it { is_expected.to contain_package('httpd') } it { is_expected.to contain_user("apache") } end end

Rspec Run [rnelson0@build03 profile:production]$ bundle exec rspec spec/classes/apache_spec.rb profile::apache with defaults for all parameters should contain Class[profile::apache] should contain Package[httpd] should contain User[apache] Finished in 7.82 seconds (files took 2.49 seconds to load) 3 examples, 0 failures

Refactor •  Create a new branch for the target version, e.g. 3.8.7

•  Test against current and target versions, e.g. ~>3.6.0 and ~>3.8.0, with and without Future Parser/Strict Variables

•  Identify failing tests, refactor to fix

•  Upgrade modules as early as possible. Be aware of the required Puppet version for a module version, and look out for defunct or migrated modules, such as those transferred to Vox Pupuli

•  Move forward when tests are green for current AND next version

Testing with particular Puppet versions

$ grep PUPPET Gemfile gem "puppet", ENV['PUPPET_GEM_VERSION'] || '~> 4.0'

[rnelson0@build controlrepo]$ export PUPPET_GEM_VERSION='~>3.8.0' [rnelson0@build controlrepo]$ bundle update Installing puppet 3.8.7 (was 4.6.0) [rnelson0@build controlrepo]$ bundle exec puppet --version 3.8.7

[rnelson0@build controlrepo]$ export PUPPET_GEM_VERSION='3.8.1' [rnelson0@build controlrepo]$ bundle update Installing puppet 3.8.1 (was 3.8.7) [rnelson0@build controlrepo]$ bundle exec puppet --version 3.8.1

High level Master(s) upgrade process •  Prep for new master/in-place upgrade

•  Deploy new/upgrade in testing

•  Revert

•  Deploy new/upgrade in production

•  Start with Master of Masters or other “parent” nodes first

•  Update separate PuppetDB node, puppetdb-termini on masters

Replace the Master •  Prepare a new operational environment

•  Do not serve bad/incorrect catalogs to existing nodes

•  Deploy a new master on the target puppet version

•  Bootstrap configuration/code

•  Test the master against itself, puppet agent -t

•  Deploy and test canary nodes in the same operational environment

In-place Master upgrade •  Snapshot (or equivalent) the master(s) and canary nodes

•  Restrict access to the master:

•  Control access with firewall/load balancer

•  Disable puppet agent on nodes with orchestration

•  Revoke certificates for non-canary nodes

•  Revoke the CA, generate a new CA and new agent certs for canary nodes only

•  Upgrade the master

•  Test the master then canary nodes with puppet agent -t

Troubleshooting •  Collect logs from the master and canaries

•  Look for changed resources, number of resources in catalog

•  Revert production environment

•  Analyze cause(s)

•  Refactor code and data to address issues

•  Try again

•  Learn from failures, prevent them in the future

Upgrade the Agents •  Can often skip on PATCH versions and some MINOR versions (see rel notes)

•  puppetlabs/puppet_agent (requirements) updates agents on next check-in

•  Orchestration

•  Replace nodes with new instances running the new agent

•  By hand

Repeat •  Relax, enjoy the success of an upgrade!

•  Start working on the next version/feature flags

•  Repeat the Refactor / Upgrade steps

Keeping Up

Keeping Up Refactor to take advantage of Puppet 4 language improvements, new tools (ex: r10k -> PE Code Manager), new file locations, etc.

•  PE has quarterly upgrades, FOSS more frequent

•  The less frequently you do something, the more painful it is. “Upgrade early and upgrade often!”

•  Try not to get more than 2 MINORs behind

•  Test against puppet version ~>4.0 (latest v4) and run bundle update before manual tests

Puppet 4 Language Improvements •  Replace create_resources() with iteration

•  Replace validate_*() with data types (including a Sensitive type)

•  There is a validate_legacy() helper function available in puppetlabs/stdlib to assist with replacing validate_*() functions (blog)

•  Simplified resource wrappers with * and + operators

•  Improved default attributes are per-expression

•  New template type EPP is available

•  Puppet Lookup, Data In Modules, and other hiera improvements

•  Use $facts[] instead of global variables to tidy up the namespace and remove ambiguity

Tips & Tricks – Puppet Enterprise •  PE includes support, use it for planning/errors

•  Puppet Enterprise Upgrade Service to engage Pro Services

•  PE Classifier changes over time. Review Preconfigured Node Groups documentation

•  pe_puppetserver_gem is out, puppetserver_gem is in

•  Do not use PE’s bundled Ruby for other Ruby tasks, conflicts between bundled/downloaded gems. Recommend rbenv/rvm or SCL-equiv instead

•  Do not ever do this on your master. EVER!

Tips & Tricks - Strings Understand how string conversion works in puppet, hiera, rspec-puppet, and how it has changed:

•  rspec-puppet: 'undef' represents an undefined value

•  Puppet DSL: it is the string undef! Try :undef, without quotes, instead

•  If you have a file resource with a title or path of ${undefvar}/${populatedvar}, rspec will start failing because file { 'undef/etc/app.conf' :} is not valid

•  Similar issue with 'true' vs true and 'false' vs false

•  Other common issues: input from hiera/ENC, quoted numbers as strings, stringify vs structured facts, unquoted strings in case selectors, etc

•  May require acceptance tests/canary nodes to become apparent

Tips & Tricks - Hiera •  Hiera eyaml gem is lost during the upgrade to the 4.x puppetserver

•  Enable the yaml backend and ensure that the master does not rely on eyaml data

•  Run the agent on the master to redeploy the gem (with puppet/hiera or similar) before agents check in

•  %{}: used to prevent variable interpolation, as in %%{}{environment} to generate the string %{environment}. In 3.x and in 4.5 resolves to an empty string, in 4.0-4.4 it returned the scope, giving strings like %<#Hiera:7329A802#>{environment}. Use %{::} instead, as in %%{::}{environment}. Affects PE < 2016.2.0

•  datadir: some versions expect :: prepends to variables and others do not. Change %{environment} to %{::environment}. Likely PE < 2016.2.0 as well

Tips & Tricks - Other •  Review modules and their supported versions. May be incorrect or weak

assumptions (>= 3 but should also include < 4 – check tests)

•  Upgrades across major versions mean additional troubleshooting

•  Upgrade early – but with caveats

•  Many tools to assist with automating version upgrades in your Puppetfile

•  ERB scope: prepend most variables with @ (<%= var %> to <%= @var %>)

•  Script to detect usage of hardcoded /etc/puppet paths, no longer correct in v4

•  External fact weighting bug: FACT-1413

•  Minimize coupled/entangled changes

•  Ask for help! Colleagues, social media, etc.

Tools •  Puppet Community Slack / IRC and Mailing Lists

•  puppet-ghostbuster helps you find "dead code" that you may want to prune before you start on your refactoring journey.

•  rspec-puppet, puppetlabs_spec_helper, and puppet-lint are improving their Puppet 4 support

•  A number of catalog diff tools exist (diff generators and a viewer) to inspect the actual catalog differences from active nodes across different versions of Puppet.

Links Additional information on Puppet 4 and Migrations

•  Official Puppet Upgrade Docs

•  Whirlwind Tour of Puppet 4 by R.I. Pienaar

•  The Power of Puppet 4 by Martin Alfke

•  Puppet - our journey from Puppet 3.8 to Puppet 4 by Jonas Genannt

Summary •  Plan the upgrade blueprint

•  Have working tests before upgrading

•  Step through the new versions / feature flags

•  Refactor code to take advantage of the language and tool improvements

•  Keep mowing

•  Enjoy the journey!