Puppet Data Mining

Puppet Data MiningReport processors, stored configs and more

PuppetCamp 22nd March 2012

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/map408/2412123378

Gareth Rushgrove @garethr

gareth rushgrove | morethanseven.net

Blog at morethanseven.net

Curate devopsweekly.com

Work at UK Government Digital Service

Serious Government Business

http://www.flickr.com/photos/iancarroll/5027441664gareth rushgrove | morethanseven.net

Puppet data

- Why do we want it?- How do we get our hands on it- What can we build with it

Puppet data

Single source of truth

http://www.flickr.com/photos/joeshlabotnik/4132307070/

Example: find a list of hostnames

- Spreadsheets- Wiki- Deployment scripts- Application code- SSH configs- More deployment scripts- Monitoring configuration

http://www.flickr.com/photos/nettsu/5276327339gareth rushgrove | morethanseven.net

Operational reports

Example: hardware spreadsheet

Company hardware

Hooks for tooling

http://www.flickr.com/photos/nickstone333/3135318558

Example: Nagios integration

Example: Rundeck integration

Where to find the data

- Report processors- Stored configurations- Puppet internal APIs

http://www.flickr.com/photos/nathaninsandiego/3757033518gareth rushgrove | morethanseven.net

Report processors

[agent]report = truepluginsync = true [master]reports = storepluginsync = true

Configuration

[agent]report = truepluginsync = true [master]reports = store,http,log,samplepluginsync = true

Adding reports

Built in

Store the YAML report on disk. Each host sends its report as a YAML dump and this just stores the file on disk, in the reportdir directory.

Send all received logs to the local log destinations. Usually the log destination is syslog.

Send report information via HTTP to the reporturl. Each host sends its report as a YAML dump and this sends this YAML to a client via HTTP POST.

Tagmail

This report sends specific log messages to specific email addresses based on the tags in the log messages.

RRDgraph

Graph all available data about hosts using the RRD library.

Simple report structure

.└── sample-report ├── lib │ └── puppet │ └── reports │ └── sample.rb └── manifests └── init.pp

Simple report processor

require 'puppet'require 'yaml'require 'logger'

LOG = Logger.new('/tmp/puppet.log')

Puppet::Reports.register_report(:sample) do def process message = "Puppet run for #{self.host} #{self.status}" Puppet.debug "Hello from sample report processor" LOG.info message endend

Register report

Do something with the data in self

Open source examples

http://www.flickr.com/photos/nettsu/5771348892

IRC/Campfire

Ganglia

Graphs!

Zendesk

James Turnbull will have got there first

Stored configuration database

Configuration

[master]storeconfigs = truedbadapter = mysqldbuser = puppetdbpassword = passworddbserver = localhostdbsocket = /var/run/mysql.sock

http://www.flickr.com/photos/nettsu/4997777753

mysql> SHOW TABLES;+------------------+| Tables_in_puppet |+------------------+| fact_names || fact_values || hosts || inventory_facts || inventory_nodes || param_names || param_values || puppet_tags || resource_tags || resources || source_files |+------------------+11 rows in set (0.00 sec)

Lots of tables to explore

Easily get a list of hosts

mysql> SELECT filename,updated_at FROM source_files;+-----------------------------------------------------+---------------------+| filename | updated_at |+-----------------------------------------------------+---------------------+| NULL | 2011-10-11 10:34:36 || /etc/puppet/modules/users/manifests/init.pp | 2011-10-12 14:13:49 || /etc/puppet/modules/nagios/manifests/init.pp | 2011-10-12 14:13:49 || /etc/puppet/modules/ntp/manifests/init.pp | 2011-10-12 14:13:50 || /etc/puppet/modules/base_packages/manifests/init.pp | 2011-10-12 14:13:50 || /etc/puppet/modules/sudo/manifests/init.pp | 2011-10-12 14:13:50 || /etc/puppet/modules/apt/manifests/init.pp | 2011-10-12 14:13:50 || /etc/puppet/modules/logrotate/manifests/init.pp | 2011-10-12 14:13:51 || /etc/puppet/modules/ganglia/manifests/init.pp | 2011-10-12 14:13:52 || /etc/puppet/modules/motd/manifests/init.pp | 2011-10-12 14:13:57 || /etc/puppet/manifests/classes.pp | 2011-10-12 14:17:25 || /etc/puppet/modules/mysql/manifests/init.pp | 2011-10-12 14:17:25 || /etc/puppet/modules/apache2/manifests/init.pp | 2011-10-12 14:17:25 || /etc/puppet/modules/passenger/manifests/init.pp | 2011-10-12 14:17:26 |+-----------------------------------------------------+---------------------+

Other interesting data too

Active Record

http://www.flickr.com/photos/simplebitsdan/480696708

#!/usr/bin/env ruby

require 'puppet/rails'

Puppet[:config] = '/etc/puppet/puppet.conf'Puppet.parse_configconfig = Puppet.settings.instance_variable_get(:@values)master_config = config[:master]

Load Puppet master configuration

adapter = master_conf[:dbadapter]args = {:adapter => adapter, :log_level => master_conf[:rails_loglevel]}

args[:host] = master_conf[:dbserver]args[:username] = master_conf[:dbuser]args[:password] = master_conf[:dbpassword]args[:database] = master_conf[:dbname]args[:database] = "puppet" unless not args[:database].to_s.empty?args[:port] = master_conf[:dbport]socket = master_conf[:dbsocket]args[:socket] = socket unless socket.to_s.empty?

Map database connection settings

ActiveRecord::Base.establish_connection(args)

Puppet::Rails::Host.all.each { |h| puts "#{h.name}"}

Make an active record query

Example run

⚡ ./list.rbubuntu.localdomain

Code spelunking

#!/usr/bin/env ruby

require 'puppet'Puppet[:config] = "/etc/puppet/puppet.conf"Puppet.parse_configPuppet[:clientyamldir] = Puppet[:yamldir]Puppet::Node.indirection.terminus_class = :yaml

Load Puppet master configuration

nodes = Puppet::Node.indirection.search("*")nodes.each do |n| facts = Puppet::Node::Facts.indirection.find(n.name) tags = Puppet::Resource::Catalog.indirection.find(n.name).tags puts "#{n.name} - #{tags.join(', ')}"end

Make a query using the internal API

Needs root permissions

⚡ sudo ./puppettags.rbubuntu.localdomain - settings, default, node

Command line tools

⚡ ./puppetlast.rb+-----------+-----------------+| hostname | last puppet run |+-----------+-----------------+| hostname4 | 100 minutes || hostname1 | 25 minutes || hostname5 | 10 minutes || hostname2 | 15 minutes || hostname3 | 5 minutes |+-----------+-----------------+

Dashboards

web-puppet

JSON over HTTP

capistrano-puppet

require 'capistrano-puppet'

web_puppet = CapistranoPuppet::Server.new( 'http://username:password@localhost:9295')

role :web doweb_puppet.get_servers('webserver')

Get hosts from puppet

Visualisation

Your imagination

- web-puppet - https://github.com/garethr/web-puppet- capistrano-puppet - https://github.com/garethr/capistrano-puppet- puppet-ganglia - https://github.com/jamtur01/puppet-ganglia- puppet-zendesk - https://github.com/jamtur01/puppet-zendesk- puppet-irc - https://github.com/jamtur01/puppet-irc- puppet-campfire - https://github.com/jamtur01/puppet-campfire

http://www.flickr.com/photos/benterrett/6852348725/

One more thing

Questions?

gareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/

Puppet Data Mining

Technology

Transcript of Puppet Data Mining

Building Data-Driven Infrastructure with Puppet - PuppetConf 2013

Puppet Camp Paris 2016 Data in Modules

Downloads.puppetlabs.com Puppet Puppet

Data Mining Introduction to Data Mining

Data Mining-Graph Mining

Cassandra & puppet, scaling data at $15 per month

Hacking The Data out of Puppet - PuppetConf '12

Helping Data Teams with Puppet / Puppet Camp London - Apr 13, 2015

PuppetConf. 2016: External Data in Puppet 4 – R.I. Pienaar

Getting Started with Puppet by Chad Metcalf Wibi Data

UNIT - I Data Mining. UNIT - I Introduction : Fundamentals of data mining, Data Mining Functionalities, Classification of Data Mining systems, Major issues.

Data Mining vs. Statistics Pavel Brusilovsky. 2 Objectives 2 Intro to Data Mining Data Mining vs. Statistics Data Mining vs. Text Mining Applications.

Introduction to Data Mining - homepages.math.uic.eduhomepages.math.uic.edu/~jyang06/stat486/R/DataMining_JYang2014.pdfFundamentals of Data Mining Typical Data Mining Tasks Data Mining

Encrypting sensitive data for puppet

Gufran Ahmad. Contents What is Data Mining? Data Mining / KDD process Different aspects of Data Mining Why Data Mining? Data Mining in Business Examples.

Puppet in Dell Networkingi.dell.com/.../data-sheets/en/Documents/puppet-in-dell-networking.pdf · Puppet in Dell Networking v1.2 6 With Open Automation Starter Kit VM: S4810-2(conf)#

MINING EDUCATIONAL DATA USING DATA MINING … · Educational data mining includes machine learning and data mining techniques. Data related to ... , density based, grid based, model

Lecture 2: Data Mining 1. Roadmap What is data mining? Data Mining Tasks – Classification/Decision Tree – Clustering – Association Mining Data Mining.

Applied Data Mining - Lagout Mining/Applied Data Mining [Xu... · The book reviews applied data mining from theoretical basis to ... data mining application arenas, ... research topics

External Data in Puppet 4