Download - Hacking The Data out of Puppet - PuppetConf '12

Dan Bode| Puppet Labs [email protected]

Hacking The Data out of Puppet

# puppetconf # puppetize @ puppetlabs

Who is this talk for?

•  People who already understand Puppet

•  Developers or people who are dev-curious


Shameless plug


What is it about?

•  Deconstructing Puppet to data

Dissecting a Puppet Run

Puppet as Data


Facter, who am I?

Agent

Hi! your facts are: kernel=linux ipaddress=10.0.0.3 macaddress=…


Hi Mr. Master, I need a catalog. Here

are my facts

http://www.dgcomputers.org/testimonials.php

Agent

facts


facts

Agent Thanks for you facts. I’ll just store them in

PuppetDB

PuppetDB


PuppetDB

facts

ENC

Agent

Mr. ENC, is this host defined as an external

node? Yep, he should be an

apache server. Here is the definition

nodes


PuppetDB

facts

Agent Just compiled your

catalog. One sec while I store it in PuppetDB.

catalog


PuppetDB

facts

Agent

catalog

Here is your catalog. Send me a report and let me know how it

went!

catalog


PuppetDB

facts

Agent

catalog

I hate to be a bother, but can you compute

the md5sums of a few files?

catalog


PuppetDB

facts

Agent Just finished applying. Here are the results.

report

catalog

Interacting with Puppet’s Data

Termini and the indirector


facts find from terminus facter

Agent


catalog find from terminus rest

http://www.dgcomputers.org/testimonials.php

Agent

facts


facts

Agent facts save to terminus

puppetdb

PuppetDB


PuppetDB

facts

ENC

Agent

node find from terminus exec (or

ldap)

nodes


PuppetDB

facts

Agent catalog find from

terminus compiler

catalog


PuppetDB

facts

Agent catalog save to

terminus puppetdb

catalog


Facter ENC

Disecting a Puppet Run

Com

piler

Config Catalogs

Nodes/Manifest

Reports

Facts


CLI commands

puppet facts find

puppet node find

puppet catalog find


CLI Puppet Facts

# mkdir –p /tmp/yaml/facts

# puppet facts find node_name --render-as yaml \

> /tmp/yaml/facts/node_name.yaml


Creating a node (optional):

# puppet node find node_name \

--node_terminus=exec \

--external_nodes=/etc/puppet/nodes.sh \

--facts_terminus=yaml \

--clientyamldir=/tmp/yaml/ --render-as=yaml \

> /tmp/yaml/nodes/node_name.yaml


Creating a catalog:

# puppet catalog find node_name \

--facts_terminus=yaml \

# puppet catalog find node_name \

--node_terminus=yaml \

--manifest=/etc/puppet/manifest/site.pp \

--modulepath=/etc/puppet/modules/

--clientyamldir=/tmp/yaml/ > /tmp/catalog.yaml


Fun with IRB

Puppet::Node::Facts.indirection.find facts

Puppet::Node.new nodes

Puppet::Catalog.indirection.find catalog


IRB Facts

irb:> require ‘puppet/face’

> facts=Puppet::Face[:facts, :current].find('node’)


Access a Fact value (irb):

…

> facts.values['ipaddress']

=> "10.0.2.15"


Creating a node (from irb):

…

> node=Puppet::Node.new('node_name',

{:classes => {:foo => {:bar => :baz}}})

>node.merge(facts.values)


Creating a catalog:

…

irb> catalog=Puppet::Resource::Catalog.indirection.\

find('node_name’, :node => node)


Use Cases


Inspecting the catalog:

•  What types are in the catalog?

irb> catalog.resources.collect do |r| r.type end.uniq

•  Gimme a resource:

irb>catalog.resource(‘Package[httpd]’)


Rspec Puppet:

let :facts do

{:operatingsystem => ‘Redhat’}

end

let :params do

{:bind_address => ‘0.0.0.0’

end

it { should contain_file(‘/tmp/foo.conf’) }


Thundering Herd

Pre-compile catalogs for faster auto-scaling


Applying pre-compiled catalogs:

puppet apply --catalog /tmp/catalog.json –server

puppet-fileserver


DMZ

tcp over USB


Use Cases


Hacking reports

Everything in Puppet is a state transition

User[‘dan’] : absent -> present

User[‘dan’][‘shell’] -> ‘/sbin/nologin’ -> /bin/bash


Setting up the agent:

[agent]

report=true


Archive reports in your yamldir

[master]

reports = store


Puppet reports

$ cd `puppet config print reportdir`

$ ls

node1 node2 node3

$ ls node1


Every report from every run ever

$ ls node1

201206060256.yaml 201206060303.yaml 201206060519.yaml 201206122349.yaml 201206122354.yaml 201206130002.yaml


Lets crack one open!

Irb > require ‘yaml’

>reports=YAML.load_file('201206130002.yaml')


Have a look

>(reports.methods - Object.methods).sort

Notice the following methods:


High level data

> reports.exit_status

⇒ 0

> reports.status

=> "unchanged"

> reports.host

⇒ ”node1”


metrics

> reports.metrics.keys

⇒ ["resources", "events", "changes", "time"]

> reports.metrics['resources']

⇒ [‘failed’, 0],[ ‘changed’, ‘7’]


And the awesome sauce

> reports.resource_statuses.keys

=> ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe-puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]


And the awesome sauce

> status = reports.resource_statuses

> status.keys

=> ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe-puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]


And the awesome sauce >events = status["File[/etc/swift/swift.conf]"].events

> events.first.status

⇒ "success”

> events.first.desired_value

⇒ :present

> events.first.previous_value

=> :absent

Thank You Dan Bode| Puppet Labs [email protected]