Puppet Performance Profiling - CM Camp 2015
Transcript of Puppet Performance Profiling - CM Camp 2015
R.I.Pienaar
February 2015
Puppet PerformanceProfiling
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Who am I?
• Puppet user since 0.22.x
• Blog at http://devco.net
• Tweets at @ripienaar
• Volcane on IRC
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Why?
• Unit and Integration testing
• Cloud based scaling
• Maintenance windows and fast feedback
• Resource usage on the master
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Agent Life Cycle
• Agent gather facts using facter *
• Agent submit facts and request catalog
• Master compiles catalog, sends to agent *
• Agent stores catalog
• Agent applies catalog *
• Agent submits report
• Master processes report
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Shows per fact timings
$ facter —timing lsbdistid: 56.06ms operatingsystem: 57.21ms osfamily: 58.54ms macaddress: 0.29ms ipaddress6: 18.36ms ipaddress6_lo: 31.41ms mtu_dummy0: 29.82ms vlans: 0.20ms selinux: 11.91ms mtu_ip_vti0: 29.09ms
Timing Facts
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Shows per fact timings…but only the setcode part
FREEGEOURL = “http://freegeoip.net/json/"
geoipinfo = JSON.parse(open(FREEGEOURL).read)
geoipinfo.each do |k, v| next if k == "ip" Facter.add("geo_#{k.downcase}") { setcode { v } } end
Timing Facts
geo_latitude: 0.02ms geo_region_name: 0.04ms geo_metro_code: 0.04ms geo_time_zone: 0.03ms geo_country_code: 0.02ms geo_country_name: 0.02ms geo_zip_code: 0.02ms geo_region_code: 0.04ms geo_city: 0.04ms geo_longitude: 0.03ms
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Restructuring fact to do the slow work in setcode
geoipinfo = nil
["country_code", “country_name”…].each do |key| Facter.add(“geo_%s” % key.downcase) do setcode do geoipinfo ||= JSON.parse(open(FREEGEOURL).read)
["", nil].include?(geoipinfo[key]) ? "unknown" : geoipinfo[key] end end end
Timing Facts
geo_zipcode: 7174.49ms geo_metro_code: 0.04ms geo_country_code: 0.03ms geo_region_name: 0.04ms geo_latitude: 0.05ms geo_area_code: 0.04ms geo_country_name: 0.03ms geo_city: 0.04ms geo_longitude: 0.04ms geo_region_code: 0.04ms
Fetch once per runand re-use
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Add a local cache, now only the first invoke is slow
def geoip_cached_fetch store = PStore.new(File.join(Dir.tmpdir, “fgeoip_fact.pstore")) store.transaction do store[:freegeoip] ||= JSON.parse(open(FREEGEOURL).read) end end
geoipinfo = nil
["country_code", “country_name”…].each do |key| Facter.add(“geo_%s” % key.downcase) do setcode do geoipinfo ||= geoip_cached_fetch
["", nil].include?(geoipinfo[key]) ? "unknown" : geoipinfo[key] end end end
Timing Facts
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Catalog Compilation
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
class apache($version=present) { validate_string($version)
class{“::apache::package”: version => $version } }
Compilation
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
# /etc/puppet/modules/apache/init.pp class apache($version=present) { validate_string($version)
class{“::apache::package”: version => $version } }
Compilation
Needs to find the file
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
class {'epel': } -> class {'puppetdb': } -> class {'puppet::master': storeconfigs => true }
Compilation
$ strace puppet parser validate test.pp 2>&1 |grep stat|wc -l 10546
$ strace puppet parser validate test.pp 2>&1 |egrep '((stat|open).+modules)'|wc -l 217
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
# /etc/puppet/modules/apache/init.pp class apache($version=present) { validate_string($version)
class{“::apache::package”: version => $version } }
Compilation
hiera(“apache::version”, “present”)
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
:hierarchy: - 01_Nodes/%{::fqdn} - 02_Domain/%{::domain} - 03_Perimeter/%{::perimeter}/%{::datacenter} - 03_Perimeter/%{::perimeter} - 04_DC/%{::datacenter} - 05_OS/%{::operatingsystem}/%{::hardwareisa} - 05_OS/%{::operatingsystem} - 06_Manufacturer/%{::manufacturer} - 07_RealTime/%{::real_time} - 09_Mco_Teams/%{::team} - 50_Teams/%{team}/01_Nodes/%{::fqdn} - 50_Teams/HPCE/%{::hpce_env}/%{::hpce_type} - 50_Teams/HPCE/%{::hpce_env}/common - 50_Teams/%{::team}/common - common
Compilation
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
class {'epel': } -> class {'puppetdb': } -> class {'puppet::master': storeconfigs => true }
Compilation
$ puppet apply test.pp 2>&1|grep "Looking for data source"|wc -l 3562
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
# /etc/puppet/modules/apache/init.pp class apache($version=present) { validate_string($version)
class{“::apache::package”: version => $version } }
Compilation
ruby code, similar issues as the fact
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
# /etc/puppet/modules/apache/init.pp class apache($version=present) { validate_string($version)
class{“::apache::package”: version => $version } }
Compilation
also speeds up finding the class internally
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
# /etc/puppet/modules/apache/init.pp class apache($version=present) { validate_string($version)
class{“::apache::package”: version => $version } }
Compilation
No hiera lookup, authoritative value supplied
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
class {'epel': } -> class {'puppetdb': } -> class {'puppet::master': storeconfigs => true }
Profiling Compilation
$ puppet apply test.pp —profile
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
importing ‘..epel/manifests/init.pp’ importing ‘..epel/manifests/params.pp’ importing ‘..epel/manifests/rpm_gpg_key.pp’ importing ‘..puppetdb/manifests/init.pp’ importing ‘..puppetdb/manifests/params.pp’ PROFILE [apply] 2.3.1 Called defined: took 0.0001 seconds PROFILE [apply] 2.3.2 Called downcase: took 0.0000 seconds PROFILE [apply] 2.3.3 Called validate_re: took 0.0000 seconds PROFILE [apply] 2.3.4 Called downcase: took 0.0000 seconds PROFILE [apply] 2.3.5 Called validate_re: took 0.0000 seconds PROFILE [apply] 2.3.6 Called downcase: took 0.0001 seconds PROFILE [apply] 2.3.7 Called validate_re: took 0.0000 seconds importing ‘..puppetdb/manifests/server.pp’ in environment production PROFILE [apply] 2.3.8 Called downcase: took 0.0000 seconds PROFILE [apply] 2.3.9 Called validate_re: took 0.0000 seconds
Profiling Compilationclass {'epel': } -> class {'puppetdb': } -> class {'puppet::master': storeconfigs => true }
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Scope(Apache::Vhost[default]): Retrieving template apache/vhost.conf.erb
template[…vhost.conf.erb]: Bound template variables for …vhost.conf.erb in 0.00 seconds
template[…vhost.conf.erb]: Interpolated template …vhost.conf.erb in 0.05 seconds
Profiling Compilationdefine apache::vhost(…) { file { “${priority_real}-${filename}.conf": content => template(‘apache/vhost.conf.erb’) } }
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
AGGREGATE PROFILING RESULTS: compiler -> compile: 5.800916 ms (1 calls) compiler -> evaluate_resource: 1.53646 ms (79 calls) compiler -> evaluate_resource -> Apache::Vhost[…]: 0.148923 ms functions: 1.892293 ms (562 calls) functions -> include: 1.091945 ms (41 calls) functions -> template: 0.754388 ms (137 calls)
Profiling Compilation
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Catalog apply phase
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
# puppet agent —test —evaltrace Class[Postgresql::Server::Reload]: Starting to evaluate Class[Postgresql::Server::Reload]: Evaluated in 0.00 seconds Class[Postgresql::Server::Config]: Starting to evaluate Class[Postgresql::Server::Config]: Evaluated in 0.00 seconds Class[Postgresql::Server::Service]: Starting to evaluate Class[Postgresql::Server::Service]: Evaluated in 0.00 seconds Apache::Vhost[puppet-localhost]: Starting to evaluate Apache::Vhost[puppet-localhost]: Evaluated in 0.00 seconds Package[mailcap]: Starting to evaluate Package[mailcap]/ensure: created Package[mailcap]: Evaluated in 65.36 seconds
Profiling the Agent
Tracing the catalog apply phase
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
# report_print.rb —count 5 —report 201411101110.yaml Report for localhost at Wed Nov 10 11:10:54 +0000 2014
Report File: /var/…/last_run_report.yaml Report Kind: apply Puppet Version: 3.7.3 Report Format: 4 Configuration Version: 1416395447 UUID: d4a9fac9-…-7db61b0dfa89 Log Lines: 2 (show with —log)
Profiling the Agent
Processing reports for performance metrics
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Report Metrics:
Changes: Total: 2 Events: Total: 2 Success: 2 Failure: 0 Resources: Total: 213 Out of sync: 2 Changed: 2 Scheduled: 0 Skipped: 0 Failed to restart: 0 Failed: 0 Restarted: 0
Profiling the Agent
Processing reports for performance metrics
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Time: Total: 28.44 Package: 15.55 Config retrieval: 7.71 File: 1.43 Exec: 1.33 Puppetdb conn validator: 0.87 Service: 0.73 Postgresql psql: 0.66 Augeas: 0.07 Ini setting: 0.04
Profiling the Agent
Processing reports for performance metrics
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Resources by resource type:
107 File 38 Ini_setting 13 Exec 12 Postgresql_psql 9 Package 6 Yumrepo 6 Anchor 6 Schedule 4 Service 3 Postgresql_conf
Profiling the Agent
Processing reports for performance metrics
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Slowest 5 resources by evaluation time:
15.55 Package[mailcap] 0.88 Puppetdb_conn_validator[puppetdb_conn] 0.66 File[/usr…/validate_postgresql_connection.sh] 0.31 Exec[validate postgres connection] 0.30 Service[puppetmaster]
Profiling the Agent
Processing reports for performance metrics
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
5 largest managed files (only those that are readable)
3.85 KB /var/…/bin/concatfragments.sh 1.65 KB /etc/puppet/rack/config.ru 1.61 KB /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6 1.23 KB /etc/puppet/puppet.conf 849.00 B /etc/httpd/conf/httpd.conf
Profiling the Agent
Processing reports for performance metrics
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Questions?
twitter: @ripienaar
email: [email protected]
blog: www.devco.net
github: ripienaar
freenode: Volcane
geo fact: http://srt.ly/hxreport processor: http://srt.ly/gq