Bloated Chefs: A Tale of Gluttony, and the Path to Enlightenment
Transcript of Bloated Chefs: A Tale of Gluttony, and the Path to Enlightenment
4/3/15
Agenda
BLOATED CHEFS
1. Chef resources in use at PD
2. Problems encountered as we grew
3. Measuring chef-client run
4. How we fixed it
5. How fast is it now?
4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs
• Awkward pauses at the beginning of the run
4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs
• Awkward pauses at the beginning of the run
• chef-client run took several minutes
4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs
• Awkward pauses at the beginning of the run
• chef-client run took several minutes
• chef-client OOM
4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs
• Awkward pauses at the beginning of the run
• chef-client run took several minutes
• chef-client OOM
4/3/15
Measuring Run Time
BLOATED CHEFS
https://github.com/joemiller/chef-handler-profiler
4/3/15
Measuring Resources
BLOATED CHEFS
• Total number of resources per run, by type
• Number of updated resources per run, by type
4/3/15
Measuring Memory
BLOATED CHEFS
• Gather proc stats with sys-proctable
• Gather GC stats
• Can be emitted as statsd
4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled
4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled
• hash2node
4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled
• hash2node
• Two searches touched
4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled
• hash2node
• Two searches touched
90s -> 60s
4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled
• hash2node
• Two searches touched
90s -> 60s30%
4/3/15
Result Memoization
BLOATED CHEFS
• Common search data
• API-backed LWRP’s
• Can be generalized
4/3/15
Result Memoization
BLOATED CHEFS
• Common search data
• API-backed LWRP’s
• Can be generalized
4/3/15
Other Nasties
BLOATED CHEFS
• Too many conditional guards
• tmpfs storage
• Multiple package resources (Chef 12)
4/3/15
Other Nasties
BLOATED CHEFS
• Too many conditional guards
• tmpfs storage
• Multiple package resources (Chef 12)
Six seconds for twelve packages