Post on 16-Feb-2017
Hiera and HieraIntroduction to hiera puppet-camp Melbourne 2015
Who am I
These really were the droids you were looking for
Sportsbet puppet history (according to me)
July 2014 a working puppet code with CICD pipeline < 50 puppetized VM’s
- Mix of physical and virtual servers.- Minimal documentation on how systems were built and configured
Nov 2014 Mobile stack built and configured with puppet.
Feb 2015 %99 percent virtualized
July 2015 ~ 900 SOE machines under puppet management
- ~ 200 machines legacy VM’s to do
AgendaTake this
$database_server = $::environment ? { 'production' => 'proddb1.puppetlabs.vm', 'development' => 'odinsraven.puppetlabs.vm', 'QA' => 'qadb1.puppetlabs.vm', 'staging' => 'stagingdb1.puppetlabs.vm',}
And turn it into this
$database_server = hiera('database_server')
Why Hiera?
“We have a module for that, but I can’t share it because it has all our company-specific data in it.”
So what is it?“Hiera is a key/value lookup tool for configuration data, built to make Puppet better and let you set node-specific data without repeating yourself.”
‘Faster-er. Easier-er. Better-er'
What should I put in Hiera:Business-specific data (i.e. internal NTP server, environment config, etc…)
Sensitive company specific data
Data that you don’t want to share with anyone else
What should I NOT put in Hiera:OS-specific data
Data that EVERYONE ELSE who uses this module will need to know (paths to config files, package names, etc…)
Backend types--:backends: - yaml:hierarchy: - hostname/"%{hostname}" - environment/"%{environment}" - common
:yaml: :datadir: /vagrant/hieradata
yaml , json , file , ldap ,gpg ,eyaml,mysql,postgres, redis
The Hierarchy--:backends: - yaml:hierarchy: - hostname/"%{hostname}" - environment/"%{environment}" - common
:yaml: :datadir: /vagrant/hieradata
Hiera likes facts and interacts with facts
Where the data lives--:backends: - yaml:hierarchy: - hostname/"%{hostname}" - environment/"%{environment}" - common
:yaml: :datadir: /vagrant/hieradata
Complete hiera Config--:backends: - yaml:hierarchy: - hostname/"%{hostname}" - environment/"%{environment}" - common
:yaml: :datadir: /vagrant/hieradata
ln -s /etc/hiera.yaml /etc/puppet/hiera.yaml
You will want to do this to avoid confusion/issues
Hierarchy hell:hierarchy: - "%{clientcert}" - "%{environment}" - "%{osfamily}" - "%{osfamily}/%{operatingsystem}" - "%{osfamily}/%{operatingsystem}/%{os_version_major}" - "%{osfamily}/%{operatingsystem}/%{os_version_minor}" # Repeat until you have 15 levels of WTF
A sample from a real hierarchy :hierarchy: - "virtual/%{::virtual}" - "hostname/%{::fqdn}" - "role/%{::role}/%{calling_module}" - "role/%{::role}" - "datacentre/%{::datacentre}" - "environment/%{::hiera_environment}/%{calling_module}" - "environment/%{::hiera_environment}" - "%{calling_module}" - passwords_fake - common
Use hiera in your modules in 2 waysexplicit:
$foo = hiera('foo', 'default string')
Automatic:
class bar ( $foo = 'default string') {}
called Data Bindings
As of Puppet 3.0.0, Puppet will now do a Hiera lookup for the fully namespaced value of a class parameter
Let's get practical
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
puppet apply --modulepath ./puppet/modules puppet/manifests/site.ppNotice: Scope(Class[Democlass]): Running with param1: Welcome to Puppet Camp :Notice: Scope(Class[Democlass]): Running db server : No DB for you! :
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
Add some direct hiera lookuppuppet apply --modulepath ./puppet/modules puppet/manifests/site.ppNotice: Scope(Class[Democlass]): Running with param1: Welcome to Puppet Camp :Notice: Scope(Class[Democlass]): Running db server : cinderella :
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
cat hieradata/common.yamldb_server: 'cinderella'cat hieradata/common.yamldb_server: 'cinderella'
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
Add some hiera data bindingpuppet apply --modulepath ./puppet/modules puppet/manifests/site.ppNotice: Scope(Class[Democlass]): Running with param1: Welcome to the ball :Notice: Scope(Class[Democlass]): Running db server : cinderella :
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
Detour - facter# facterarchitecture => x86_64augeasversion => 1.0.0blockdevice_sda_model => VBOX HARDDISKblockdevice_sda_size => 21474836480blockdevice_sda_vendor => ATAblockdevices => sdafacterversion => 2.4.4filesystems => ext4,iso9660fqdn => localhost.sportsbet.com.augid => vagranthardwareisa => x86_64hardwaremodel => x86_64hostname => localhostinterfaces => eth0,loipaddress => 10.0.2.15ipaddress_eth0 => 10.0.2.15ipaddress_lo => 127.0.0.1
Add some fact based data bindingpuppet apply --modulepath ./puppet/modules puppet/manifests/site.ppNotice: Scope(Class[Democlass]): Running with param1: Dude where is my slipper :Notice: Scope(Class[Democlass]): Running db server : cinderella :
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/hostname/localhost.yamldemoclass::param1: 'Dude where is my slipper'
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/hostname/localhost.yamldemoclass::param1: 'Dude where is my slipper'
Add some custom facts puppet apply --modulepath ./puppet/modules puppet/manifests/site.ppNotice: Scope(Class[Democlass]): Running with param1: Dude where is my slipper :Notice: Scope(Class[Democlass]): Running db server : Dr Martin :
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/hostname/localhost.yamldemoclass::param1: 'Dude where is my slipper'
cat hieradata/environment/dev.yamldemoclass::param1: 'Devs be free'db_server: 'Dr Martin'cat /etc/facter/facts.d/environment.yaml
---environment: dev
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/hostname/localhost.yamldemoclass::param1: 'Dude where is my slipper'
cat hieradata/environment/dev.yamldemoclass::param1: 'Devs be free'db_server: 'Dr Martin'
Importance of hierarchy order:hierarchy: - defaults - "hostname/%{hostname}" - "environment/%{environment}" - common
Add a some custom facts (fixed order) puppet apply --modulepath ./puppet/modules puppet/manifests/site.ppNotice: Scope(Class[Democlass]): Running with param1: Devs be free :Notice: Scope(Class[Democlass]): Running db server : Dr Martin :
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/hostname/localhost.yaml#democlass::param1: 'Dude where is my slipper'
cat hieradata/environment/dev.yamldemoclass::param1: 'Devs be free'db_server: 'Dr Martin'
cat /etc/facter/facts.d/environment.yaml---environment: dev
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/hostname/localhost.yaml#democlass::param1: 'Dude where is my slipper'
cat hieradata/environment/dev.yamldemoclass::param1: 'Devs be free'db_server: 'Dr Martin'
Detour - additional hiera config:merge_behavior
- native (default)- deep - deeper
Debugging hierahiera -d# hiera -d democlass::param1DEBUG: Sat Nov 14 04:27:29 +0000 2015: Hiera YAML backend startingDEBUG: Sat Nov 14 04:27:29 +0000 2015: Looking up democlass::param1 in YAML backendDEBUG: Sat Nov 14 04:27:29 +0000 2015: Looking for data source commonDEBUG: Sat Nov 14 04:27:29 +0000 2015: Found democlass::param1 in commonWelcome to the ball
# hiera -d db_serverDEBUG: Sat Nov 14 04:27:59 +0000 2015: Hiera YAML backend startingDEBUG: Sat Nov 14 04:27:59 +0000 2015: Looking up db_server in YAML backendDEBUG: Sat Nov 14 04:27:59 +0000 2015: Looking for data source commonDEBUG: Sat Nov 14 04:27:59 +0000 2015: Found db_server in commoncinderella
Debugging hiera passing facthiera -d
# hiera -d democlass::param1 environment=devDEBUG: Sat Nov 14 04:28:19 +0000 2015: Hiera YAML backend startingDEBUG: Sat Nov 14 04:28:19 +0000 2015: Looking up democlass::param1 in YAML backendDEBUG: Sat Nov 14 04:28:19 +0000 2015: Looking for data source environment/devDEBUG: Sat Nov 14 04:28:19 +0000 2015: Found democlass::param1 in environment/devDevs be free
# hiera -d db_server environment=devDEBUG: Sat Nov 14 04:28:59 +0000 2015: Hiera YAML backend startingDEBUG: Sat Nov 14 04:28:59 +0000 2015: Looking up db_server in YAML backendDEBUG: Sat Nov 14 04:28:59 +0000 2015: Looking for data source environment/devDEBUG: Sat Nov 14 04:28:59 +0000 2015: Found db_server in environment/devDr Martin
Debugging hiera with facter facts
# facter --yaml > /tmp/facter.yaml # hiera -d democlass::param1 -y /tmp/facter.yamlDEBUG: Sat Nov 14 04:36:28 +0000 2015: Hiera YAML backend startingDEBUG: Sat Nov 14 04:36:28 +0000 2015: Looking up democlass::param1 in YAML backendDEBUG: Sat Nov 14 04:36:28 +0000 2015: Looking for data source hostname/localhostDEBUG: Sat Nov 14 04:36:28 +0000 2015: Found democlass::param1 in hostname/localhostDude where is my slipper
Keeping secretsRecommend using hiera-eyaml (encrypted yaml)
If you use the ATOM editor
- https://atom.io/packages/hiera-eyaml
--- password: > ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBAD AFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEACvmEx6+xIWPpykTAsoFyCHb0MC +BWQPsWSm6yeJRgkWV+ZVhWwqz09nS1LMo6aKInzg2EjsDScpUFKeXvgxvV8 IF9Bes33V5ySwnlDR15UAWWkfyYxpjk8ozSTGyIWATRITSMWOZDzOZRBYFcD Mj57o5kykkurVqskYG9DDQ8xPq85t2pPPHjb0d/BzJaeqqXsnacDrylbLJau +Ohldb+s/ekBwj/iDIu2NL3KvRvn2VxYAXrgehi9VXFutN6E8yoE72XkSxfN gBdUNtha3DF9jOjdTx3b43WesqNfAwyFvZ2IRCUzV4K1ZvIDK2pA8bsQDbuP XkX7r2fSgG4CUK4jA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBo6hA3GV eKTzUgflmH0h9EgBCBe7XgD7ZVLBu3j6YhgL/I]
multiple Backends types --:backends: - eyaml - yaml:hierarchy: - hostname/"%{hostname}" - environment/"%{environment}" - common
:yaml: :datadir: /vagrant/hieradata
Issues you are going to haveComplicated Hierarchy
Latency / Load
Too many backends
Design / Architecture
Unhelpful errorsError: Could not retrieve catalog from remote server: Error 400 on SERVER: (<unknown>): found character that cannot start any token while scanning for the next token at line 1297 column 3Warning: Not using cache on failed catalogError: Could not retrieve catalog; skipping run
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `empty?' for nil:NilClass at /etc/puppet/environments/production/manifests/nodes.pp:1 on node example.makandra.deWarning: Not using cache on failed catalogError: Could not retrieve catalog; skipping run
Tips to deal with Unhelpful errorsTypical malformed yaml
Check your yaml with a linter
# cd /vagrant/hieradata/; \for i in `find . -name '*.yaml'`; \ do echo $i; \ruby -e "require 'yaml'; \YAML.parse(File.open('$i'))"; done
The PROsSeparation of data and code
keeping secrets
Backend integration to existing data stores
Make’s some conditional logic disappear
Make’s for better puppet code
The CON’sHard to figure out where things come from
hiera-yaml can only support one datadirectory
debugging
public modules and hiera still has some ways to go
Wrap upUse hiera it’s awesome
Start simple use yaml backend
Have defaults to your lookups for when lookups fail
Experiment with it
Lint check you yaml
Things you might want to look at laterMore Debugging
- Glowing Octo Ninja - https://github.com/llowder/Zwiebelschaler
hiera_array
hiera_hash
hiera_include
create_resources
Please go and upvote this bugPuppet user resource should read only from local databases
https://tickets.puppetlabs.com/browse/PUP-1913
@bigAl