Post on 10-May-2015
description
Dan Bode| Puppet Labs dan@puppetlabs.com
Hacking The Data out of Puppet
# puppetconf # puppetize @ puppetlabs
Who is this talk for?
• People who already understand Puppet
• Developers or people who are dev-curious
# puppetconf # puppetize @ puppetlabs
Shameless plug
# puppetconf # puppetize @ puppetlabs
What is it about?
• Deconstructing Puppet to data
Dissecting a Puppet Run
Puppet as Data
# puppetconf # puppetize @ puppetlabs
Facter, who am I?
Agent
Hi! your facts are: kernel=linux ipaddress=10.0.0.3 macaddress=…
# puppetconf # puppetize @ puppetlabs
Hi Mr. Master, I need a catalog. Here
are my facts
http://www.dgcomputers.org/testimonials.php
Agent
facts
# puppetconf # puppetize @ puppetlabs
facts
Agent Thanks for you facts. I’ll just store them in
PuppetDB
PuppetDB
# puppetconf # puppetize @ puppetlabs
PuppetDB
facts
ENC
Agent
Mr. ENC, is this host defined as an external
node? Yep, he should be an
apache server. Here is the definition
nodes
# puppetconf # puppetize @ puppetlabs
PuppetDB
facts
Agent Just compiled your
catalog. One sec while I store it in PuppetDB.
catalog
# puppetconf # puppetize @ puppetlabs
PuppetDB
facts
Agent
catalog
Here is your catalog. Send me a report and let me know how it
went!
catalog
# puppetconf # puppetize @ puppetlabs
PuppetDB
facts
Agent
catalog
I hate to be a bother, but can you compute
the md5sums of a few files?
catalog
# puppetconf # puppetize @ puppetlabs
PuppetDB
facts
Agent Just finished applying. Here are the results.
report
catalog
Interacting with Puppet’s Data
Termini and the indirector
# puppetconf # puppetize @ puppetlabs
facts find from terminus facter
Agent
# puppetconf # puppetize @ puppetlabs
catalog find from terminus rest
http://www.dgcomputers.org/testimonials.php
Agent
facts
# puppetconf # puppetize @ puppetlabs
facts
Agent facts save to terminus
puppetdb
PuppetDB
# puppetconf # puppetize @ puppetlabs
PuppetDB
facts
ENC
Agent
node find from terminus exec (or
ldap)
nodes
# puppetconf # puppetize @ puppetlabs
PuppetDB
facts
Agent catalog find from
terminus compiler
catalog
# puppetconf # puppetize @ puppetlabs
PuppetDB
facts
Agent catalog save to
terminus puppetdb
catalog
# puppetconf # puppetize @ puppetlabs
Facter ENC
Disecting a Puppet Run
Com
piler
Config Catalogs
Nodes/Manifest
Reports
Facts
# puppetconf # puppetize @ puppetlabs
CLI commands
puppet facts find
puppet node find
puppet catalog find
# puppetconf # puppetize @ puppetlabs
CLI Puppet Facts
# mkdir –p /tmp/yaml/facts
# puppet facts find node_name --render-as yaml \
> /tmp/yaml/facts/node_name.yaml
# puppetconf # puppetize @ puppetlabs
Creating a node (optional):
# puppet node find node_name \
--node_terminus=exec \
--external_nodes=/etc/puppet/nodes.sh \
--facts_terminus=yaml \
--clientyamldir=/tmp/yaml/ --render-as=yaml \
> /tmp/yaml/nodes/node_name.yaml
# puppetconf # puppetize @ puppetlabs
Creating a catalog:
# puppet catalog find node_name \
--facts_terminus=yaml \
# puppet catalog find node_name \
--node_terminus=yaml \
--manifest=/etc/puppet/manifest/site.pp \
--modulepath=/etc/puppet/modules/
--clientyamldir=/tmp/yaml/ > /tmp/catalog.yaml
# puppetconf # puppetize @ puppetlabs
Fun with IRB
Puppet::Node::Facts.indirection.find facts
Puppet::Node.new nodes
Puppet::Catalog.indirection.find catalog
# puppetconf # puppetize @ puppetlabs
IRB Facts
irb:> require ‘puppet/face’
> facts=Puppet::Face[:facts, :current].find('node’)
# puppetconf # puppetize @ puppetlabs
Access a Fact value (irb):
…
> facts.values['ipaddress']
=> "10.0.2.15"
# puppetconf # puppetize @ puppetlabs
Creating a node (from irb):
…
> node=Puppet::Node.new('node_name',
{:classes => {:foo => {:bar => :baz}}})
>node.merge(facts.values)
# puppetconf # puppetize @ puppetlabs
Creating a catalog:
…
irb> catalog=Puppet::Resource::Catalog.indirection.\
find('node_name’, :node => node)
Interacting with Puppet’s Data
Use Cases
# puppetconf # puppetize @ puppetlabs
Inspecting the catalog:
• What types are in the catalog?
irb> catalog.resources.collect do |r| r.type end.uniq
• Gimme a resource:
irb>catalog.resource(‘Package[httpd]’)
# puppetconf # puppetize @ puppetlabs
Rspec Puppet:
let :facts do
{:operatingsystem => ‘Redhat’}
end
let :params do
{:bind_address => ‘0.0.0.0’
end
it { should contain_file(‘/tmp/foo.conf’) }
# puppetconf # puppetize @ puppetlabs
Thundering Herd
Pre-compile catalogs for faster auto-scaling
# puppetconf # puppetize @ puppetlabs
Applying pre-compiled catalogs:
puppet apply --catalog /tmp/catalog.json –server
puppet-fileserver
# puppetconf # puppetize @ puppetlabs
DMZ
tcp over USB
Interacting with Puppet’s Data
Use Cases
# puppetconf # puppetize @ puppetlabs
Hacking reports
Everything in Puppet is a state transition
User[‘dan’] : absent -> present
User[‘dan’][‘shell’] -> ‘/sbin/nologin’ -> /bin/bash
# puppetconf # puppetize @ puppetlabs
Setting up the agent:
[agent]
report=true
# puppetconf # puppetize @ puppetlabs
Archive reports in your yamldir
[master]
reports = store
# puppetconf # puppetize @ puppetlabs
Puppet reports
$ cd `puppet config print reportdir`
$ ls
node1 node2 node3
$ ls node1
# puppetconf # puppetize @ puppetlabs
Every report from every run ever
$ ls node1
201206060256.yaml 201206060303.yaml 201206060519.yaml 201206122349.yaml 201206122354.yaml 201206130002.yaml
# puppetconf # puppetize @ puppetlabs
Lets crack one open!
Irb > require ‘yaml’
>reports=YAML.load_file('201206130002.yaml')
# puppetconf # puppetize @ puppetlabs
Have a look
>(reports.methods - Object.methods).sort
Notice the following methods:
# puppetconf # puppetize @ puppetlabs
High level data
> reports.exit_status
⇒ 0
> reports.status
=> "unchanged"
> reports.host
⇒ ”node1”
# puppetconf # puppetize @ puppetlabs
metrics
> reports.metrics.keys
⇒ ["resources", "events", "changes", "time"]
> reports.metrics['resources']
⇒ [‘failed’, 0],[ ‘changed’, ‘7’]
# puppetconf # puppetize @ puppetlabs
And the awesome sauce
> reports.resource_statuses.keys
=> ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe-puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]
# puppetconf # puppetize @ puppetlabs
And the awesome sauce
> status = reports.resource_statuses
> status.keys
=> ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe-puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]
# puppetconf # puppetize @ puppetlabs
And the awesome sauce >events = status["File[/etc/swift/swift.conf]"].events
> events.first.status
⇒ "success”
> events.first.desired_value
⇒ :present
> events.first.previous_value
=> :absent
Thank You Dan Bode| Puppet Labs dan@puppetlabs.com