What?
What?
• Chef at Etsy
What?
• Chef at Etsy
• Familiarity and Understanding
What?
• Chef at Etsy
• Familiarity and Understanding
• Critical Approach and Experimentation
What?
• Chef at Etsy
• Familiarity and Understanding
• Critical Approach and Experimentation
• Use The Source
What?
• Chef at Etsy
• Familiarity and Understanding
• Critical Approach and Experimentation
• Use The Source
• A liberal sprinkling of screwups
What?
• Chef at Etsy
• Familiarity and Understanding
• Critical Approach and Experimentation
• Use The Source
• A liberal sprinkling of screwups
• Open Sourced Goodness - We’re all here!
What?
• Chef at Etsy
• Familiarity and Understanding
• Critical Approach and Experimentation
• Use The Source
• A liberal sprinkling of screwups
• Open Sourced Goodness - We’re all here!
• [x] = http://tiny.cc/velocity2012
Opscode is Orange,Velocity is Blue.In Soviet Russia,
Cookbook writes you.
Chef at Etsy
Our Setup
Our Setup• Open Source chef server 0.10.4
Our Setup• Open Source chef server 0.10.4
• Backup to Opscode Platform
Our Setup• Open Source chef server 0.10.4
• Backup to Opscode Platform
• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)
Our Setup• Open Source chef server 0.10.4
• Backup to Opscode Platform
• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)
• KVM & lxc virts, self hosted too.
Our Setup• Open Source chef server 0.10.4
• Backup to Opscode Platform
• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)
• KVM & lxc virts, self hosted too.
• Never test in production!
Our Setup• Open Source chef server 0.10.4
• Backup to Opscode Platform
• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)
• KVM & lxc virts, self hosted too.
• Never test in production!
• Many chefs don’t spoil our soup
Our Setup• Open Source chef server 0.10.4
• Backup to Opscode Platform
• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)
• KVM & lxc virts, self hosted too.
• Never test in production!
• Many chefs don’t spoil our soup
Familiarity and Understanding
not
http://www.flickr.com/photos/photo_secessionist/5555167113/ (C) Alexander McQueen
• Insight
• Insight
• Simplicity
• Insight
• Simplicity
• Standards
Insight
Insight
• You should never have to say “I don’t know.”
Insight
• You should never have to say “I don’t know.”
• What, where, when, why, how long?
Insight
• You should never have to say “I don’t know.”
• What, where, when, why, how long?
• Easy, and I’ll show you how!
Chef Dashboard
Chef Dashboard• Chef handler sends metrics to graphite [4]
Chef Dashboard• Chef handler sends metrics to graphite [4]• git clone git://github.com/etsy/chef-handlers.git
Chef Dashboard• Chef handler sends metrics to graphite [4]• git clone git://github.com/etsy/chef-handlers.git
• Set your graphite server’s URL in graphite.rb
Chef Dashboard• Chef handler sends metrics to graphite [4]• git clone git://github.com/etsy/chef-handlers.git
• Set your graphite server’s URL in graphite.rb
• Add the following to client.rb
Chef Dashboard• Chef handler sends metrics to graphite [4]• git clone git://github.com/etsy/chef-handlers.git
• Set your graphite server’s URL in graphite.rb
• Add the following to client.rb• require "<clonedir>/graphite.rb"
graphite_handler = GraphiteReporting.newreport_handlers << graphite_handlerexception_handlers << graphite_handler
Chef Dashboard• Chef handler sends metrics to graphite [4]• git clone git://github.com/etsy/chef-handlers.git
• Set your graphite server’s URL in graphite.rb
• Add the following to client.rb• require "<clonedir>/graphite.rb"
graphite_handler = GraphiteReporting.newreport_handlers << graphite_handlerexception_handlers << graphite_handler
• Etsy dashboards framework [5]
[12:54:01] <irccat> Chef run failed on gfernandez.vm.ny4dev.etsy.com[12:54:02] <irccat> https://github.etsycorp.com/gist/384228[12:54:02] <irccat>
[12:54:07] <irccat> Chef run failed on buildtest11.ny4dev.etsy.com[12:54:07] <irccat> https://github.etsycorp.com/gist/384227[12:54:07] <irccat>
Chef irccat Alerts
Chef irccat Alerts
• Chef handler send fails to irc[4] via irccat [6]
Chef irccat Alerts
• Chef handler send fails to irc[4] via irccat [6]• git clone git://github.com/etsy/chef-handlers.git
Chef irccat Alerts
• Chef handler send fails to irc[4] via irccat [6]• git clone git://github.com/etsy/chef-handlers.git
• Set your irccat[6] server’s URL in logtoirc.rb
Chef irccat Alerts
• Chef handler send fails to irc[4] via irccat [6]• git clone git://github.com/etsy/chef-handlers.git
• Set your irccat[6] server’s URL in logtoirc.rb
• Add the following to client.rb
Chef irccat Alerts
• Chef handler send fails to irc[4] via irccat [6]• git clone git://github.com/etsy/chef-handlers.git
• Set your irccat[6] server’s URL in logtoirc.rb
• Add the following to client.rb• require "<clonedir>/logtoirc.rb"
exception_handlers << Etsy::LogToIRC.new
~ > knife node lastrun buildtest11.ny4dev.etsy.comStatus failed Elapsed Time 4.628171438 Start Time 2012-06-18 10:06:28 +0000End Time 2012-06-18 10:06:32 +0000
Recipe Action Resource Type Resource
Backtrace<snip>
ExceptionChef::Exceptions::Package: package[php] (php::buildtest line 20) had an error: Version 5.3.10-1.el5 of php not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)
~ > knife search node 'lastrun_debug_formatted_exception:Chef\:\:Exceptions\:\:Package*' -a lastrun.debug.formatted_exception
5 items found
id: masterrestore.ny4.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package: package[postgresql-server] (postgresql::server-8.3 line 1) had an error: Installed package postgresql-server-8.3.16-1PGDG_id is newer than candidate package postgresql-server-8.3.11-1PGDG_id.rhel5
id: buildtest11.ny4dev.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package: package[php] (php::buildtest line 20) had an error: Version 5.3.10-1.el5 of php not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)
<snip>
Chef lastrun Info
Chef lastrun Info
• Chef handler and knife plugin [7]
Chef lastrun Info
• Chef handler and knife plugin [7]
• gem install knife-lastrun
Chef lastrun Info
• Chef handler and knife plugin [7]
• gem install knife-lastrun
• Add the following to client.rb
Chef lastrun Info
• Chef handler and knife plugin [7]
• gem install knife-lastrun
• Add the following to client.rb• require "lastrun_update"
handler = LastRunUpdateHandler.newreport_handlers << handlerexception_handlers << handler
Chef lastrun Info
• Chef handler and knife plugin [7]
• gem install knife-lastrun
• Add the following to client.rb• require "lastrun_update"
handler = LastRunUpdateHandler.newreport_handlers << handlerexception_handlers << handler
• knife node lastrun <nodename>
Simplicity
Simplicity
• Think of yourself at 3AM!
Simplicity
• Think of yourself at 3AM!
• Please, won’t you think of the new guy?
Simplicity
• Think of yourself at 3AM!
• Please, won’t you think of the new guy?
• Minimize the logics!
Simplicity
• Think of yourself at 3AM!
• Please, won’t you think of the new guy?
• Minimize the logics!
• As few logical steps from start to finish as possible.
Simplicity - Not!Date: Mon Dec 05 2011 23:07:18 GMT+0000 (GMT)
Subject: so close to death
# Don't install v2 on search or Cent 5.6 nodes-if node[:fqdn] !~ /\b(^(preprod-)?search[0-9]{2}|ny4dev\.etsy\.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|^imgcache[0-9]{2}|imgconvert[0-9]{2}|^imgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})\b/ and node.role?("Web56") == false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false+if node[:fqdn] !~ /\b(^(preprod-)?search[0-9]{2}|ny4dev\.etsy\.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|^imgcache[0-9]{2}|imgconvert[0-9]{2}|^imgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})\b/ and node.role?("Web56") == false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false and node.role?("Auth") == false
Simplicity - Better!if node.chef_environment == "libmemcached_upgrade"
package "libmemcached" do
version "1.0.4-1"
action :install
end
<snip>
else
package "libmemcached" do
version "0.53-1.1"
action :install
end
<snip>
end
Simplicity - Complexity
Simplicity - Complexity
• Sometimes you need complex behaviour
Simplicity - Complexity
• Sometimes you need complex behaviour
• Don’t fight it, try to abstract it.
Simplicity - Complexity
• Sometimes you need complex behaviour
• Don’t fight it, try to abstract it.
• Case in point: Syslog-ng refactor
Case Study: Syslog-ng
Case Study: Syslog-ng
• 36 recipes
Case Study: Syslog-ng
• 36 recipes
• 30 versions of syslog-ng.conf
Case Study: Syslog-ng
• 36 recipes
• 30 versions of syslog-ng.conf
• 27 manually configured files in /etc/syslog-ng.d on central server
Case Study: Syslog-ng
• 36 recipes
• 30 versions of syslog-ng.conf
• 27 manually configured files in /etc/syslog-ng.d on central server
• Edge cases and exceptions galore
Case Study: Syslog-ng
Case Study: Syslog-ng
• Down to:
Case Study: Syslog-ng
• Down to:
• 2 recipes (one client, one server)
Case Study: Syslog-ng
• Down to:
• 2 recipes (one client, one server)
• 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d)
Case Study: Syslog-ng
• Down to:
• 2 recipes (one client, one server)
• 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d)
• Attributes in roles
Case Study: Syslog-ng
• Down to:
• 2 recipes (one client, one server)
• 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d)
• Attributes in roles
• Not open sourced yet, sorry :(
Case Study: Syslog-ng"syslog": {
"group": "preprod_web",
"items": {
"web_apache_access_log": {
"source": "/var/log/httpd/access_log",
"source_program_override": "APACHEACCESS: ",
"destination": "<snip>/access.log",
"destination_filters": [
"host('^preprod-web')",
"match('APACHEACCESS')"
],
"destination_options": [
"template_escape(no)"
]
},
}
}
Remember, No Panacea!
Remember, No Panacea!
• A new package hits the repo.
Remember, No Panacea!
• A new package hits the repo.
• Are you in control of when it goes out?
Remember, No Panacea!
• A new package hits the repo.
• Are you in control of when it goes out?
• Memcached Outage
Remember, No Panacea!
• A new package hits the repo.
• Are you in control of when it goes out?
• Memcached Outage
• Do you know what services are going to restart and when?
Remember, No Panacea!
• A new package hits the repo.
• Are you in control of when it goes out?
• Memcached Outage
• Do you know what services are going to restart and when?
• Image Service Outage
Standards
Standards
• Not going to talk about testing [8]!
Standards
• Not going to talk about testing [8]!
• But I don’t have time for standards!
Standards - No Time!
Standards - No Time!
• I won’t say “Make Time”, but you should...
Standards - No Time!
• I won’t say “Make Time”, but you should...
• For a quick win, try Foodcritic
Standards - No Time!
• I won’t say “Make Time”, but you should...
• For a quick win, try Foodcritic
• Good out of the box rules
Standards - No Time!
• I won’t say “Make Time”, but you should...
• For a quick win, try Foodcritic
• Good out of the box rules
• Jenkins integration in seconds
Standards - No Time!
• I won’t say “Make Time”, but you should...
• For a quick win, try Foodcritic
• Good out of the box rules
• Jenkins integration in seconds
• Supports custom rules
Standards - No Time!
• I won’t say “Make Time”, but you should...
• For a quick win, try Foodcritic
• Good out of the box rules
• Jenkins integration in seconds
• Supports custom rules
• Plays well with others
Foodcritic
Foodcritic•gem install foodcritic
Foodcritic•gem install foodcritic
•foodcritic <cookbook_repo>
Foodcritic•gem install foodcritic
•foodcritic <cookbook_repo>
•Simple Jenkins job:
Foodcritic•gem install foodcritic
•foodcritic <cookbook_repo>
•Simple Jenkins job:
#!/usr/bin/env rvm-shell 1.9.3foodcritic -f correctness .
Standards at Etsy
Standards at Etsy• “style” not “correctness”[9]
Standards at Etsy• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
Standards at Etsy• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
Standards at Etsy• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
• ETSY003 - Execute resource used to run curl or wget commands
Standards at Etsy• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
• ETSY003 - Execute resource used to run curl or wget commands
• ETSY004 - Execute resource defined without conditional or action :nothing
Standards at Etsy• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
• ETSY003 - Execute resource used to run curl or wget commands
• ETSY004 - Execute resource defined without conditional or action :nothing
• ETSY005 - Action :restart sent to a core service
Standards at Etsy• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
• ETSY003 - Execute resource used to run curl or wget commands
• ETSY004 - Execute resource defined without conditional or action :nothing
• ETSY005 - Action :restart sent to a core service
• ETSY006 - Execute resource used to run chef-provided command
Standards at Etsy• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
• ETSY003 - Execute resource used to run curl or wget commands
• ETSY004 - Execute resource defined without conditional or action :nothing
• ETSY005 - Action :restart sent to a core service
• ETSY006 - Execute resource used to run chef-provided command
• ETSY007 - Package or yum_package resource used to install core package without specific version number
Standards at Etsy• “style” not “correctness”[9]
• ETSY001 - Package or yum_package resource used with :upgrade action
• ETSY002 - Execute resource used to run git commands
• ETSY003 - Execute resource used to run curl or wget commands
• ETSY004 - Execute resource defined without conditional or action :nothing
• ETSY005 - Action :restart sent to a core service
• ETSY006 - Execute resource used to run chef-provided command
• ETSY007 - Package or yum_package resource used to install core package without specific version number
Standards at Etsy
Standards at Etsy
• ETSY001 - Written after Memcached Outage
Standards at Etsy
• ETSY001 - Written after Memcached Outage
• Package or yum_package resource used with :upgrade action
Standards at Etsy
• ETSY001 - Written after Memcached Outage
• Package or yum_package resource used with :upgrade action
•package "memcached" do action :upgradeend
Standards at Etsy
Standards at Etsy
• ETSY005 - Written after a total Image service outage
Standards at Etsy
• ETSY005 - Written after a total Image service outage
• Action :restart sent to a core service
Standards at Etsy
• ETSY005 - Written after a total Image service outage
• Action :restart sent to a core service• cookbook_file "/etc/httpd/conf.d/myvhost.conf" do
source "myvhost.conf" notifies :restart, resources(:service => "httpd")end
Critical Approach and Experimentation
CA&E
CA&E
• Chef is by necessity generic
CA&E
• Chef is by necessity generic
• ...so don’t take Opscode’s word for it.
CA&E
• Chef is by necessity generic
• ...so don’t take Opscode’s word for it.
• If it doesn’t work well for you, change it!
CA&E
• Chef is by necessity generic
• ...so don’t take Opscode’s word for it.
• If it doesn’t work well for you, change it!
• Case Study - Etsy Environments rollout
Environments Rollout
Environments Rollout
• We knew we needed them
Environments Rollout
• We knew we needed them
• Simple enough, right?
Environments Rollout
• We knew we needed them
• Simple enough, right?
• Let’s look at the workflow...
Env: Standard Workflow
Env: Standard Workflow
• knife cookbook show php
Env: Standard Workflow
• knife cookbook show php
• Change version number in metadata.rb
Env: Standard Workflow
• knife cookbook show php
• Change version number in metadata.rb
• Change version constraint in foo.json
Env: Standard Workflow
• knife cookbook show php
• Change version number in metadata.rb
• Change version constraint in foo.json
• Commit and push changes to git
Env: Standard Workflow
• knife cookbook show php
• Change version number in metadata.rb
• Change version constraint in foo.json
• Commit and push changes to git
• knife cookbook upload php --freeze
Env: Standard Workflow
• knife cookbook show php
• Change version number in metadata.rb
• Change version constraint in foo.json
• Commit and push changes to git
• knife cookbook upload php --freeze
• knife environment from file foo.json
Env: Our Issues
Env: Our Issues
• 41 people with Chef repo access
Env: Our Issues
• 41 people with Chef repo access
• Most with knife keys
Env: Our Issues
• 41 people with Chef repo access
• Most with knife keys
• All editing the same 2 files with every change...
Env: Solutions?
Env: Solutions?
• Go with it and hope for the best?
Env: Solutions?
• Go with it and hope for the best?
• Don’t use environments?
Env: Solutions?
• Go with it and hope for the best?
• Don’t use environments?
• Write a totally new workflow?
Env: Solutions?
• Go with it and hope for the best?
• Don’t use environments?
• Write a totally new workflow?
• Tweak the existing one with some tooling?
Env: Solution!
Env: Solution!
• Tweak with some tooling!
Env: Solution!
• Tweak with some tooling!
• Presenting knife-spork [10]...
Spork: Workflow
Spork: Workflow
• Wrapper around standard environments workflow
Spork: Workflow
• Wrapper around standard environments workflow
• check - cookbook versioning
Spork: Workflow
• Wrapper around standard environments workflow
• check - cookbook versioning
• bump - increment version component
Spork: Workflow
• Wrapper around standard environments workflow
• check - cookbook versioning
• bump - increment version component
• upload - upload and freeze
Spork: Workflow
• Wrapper around standard environments workflow
• check - cookbook versioning
• bump - increment version component
• upload - upload and freeze
• promote - set env constraints
Spork: Check$> knife spork check apache2
Checking versions for cookbook apache2...
Current local version: 1.0.6
Remote versions (Max. 5 most recent only):*1.0.6, frozen1.0.5, frozen<snip>
DANGER: Your local cookbook has same version number as the starred version above!
Please bump your local version or you won't be able to upload.
Spork: Bump
$> knife spork bump apache2 <major|minor|patch|manual>
Bumping patch level of the apache2 cookbook from 1.0.6 to 1.0.7
Spork: Upload
$> knife spork upload apache2
Uploading and freezing apache2 [1.0.7]upload complete
Spork: Promote$> knife spork promote foo php
Adding version constraint php = 1.0.6
Saving changes into foo.json
Promotion complete! Please remember to upload your changed Environment file to the Chef Server.
---
$> knife spork promote foo php --remoteAdding version constraint php = 0.1.0
Saving changes into foo.json
Uploading foo to server
Spork
Spork
• Worked well, avoided the issues it was designed for
Spork
• Worked well, avoided the issues it was designed for
• Subsequently evolved
Spork
• Worked well, avoided the issues it was designed for
• Subsequently evolved
• A lot of input & work came from Devs
Spork
• Worked well, avoided the issues it was designed for
• Subsequently evolved
• A lot of input & work came from Devs
• Organic evolution
Spork
• Worked well, avoided the issues it was designed for
• Subsequently evolved
• A lot of input & work came from Devs
• Organic evolution
• Open sourced, of course
Spork: Evolution
Spork: Evolution
• Safety Checks
Spork: Evolution
• Safety Checks
• Extra Features
Spork: Safety Checks
Spork: Safety Checks
• Before promoting, check version is uploaded...
Spork: Safety Checks
$> knife spork promote php --remote
<snip>
WARNING: It looks like you have multiple cookbook paths defined so I can't tell if you're running inside a git repo.
Checking that php version 0.1.93 exists on the server before promoting (any error means it hasn't been uploaded yet)...
ERROR: The object you are looking for could not be found
Response: Cannot find a cookbook named php with version 0.1.93
Spork: Safety Checks
• Before promoting, check version is uploaded...
• Check if you’re promoting changes to more than you thought....
Spork: Safety ChecksWARNING: You're about to promote changes to several cookbooks:
WARNING: ganglia: = 0.1.26 changed to = 0.1.25installerz: = 0.1.66 changed to = 0.1.65php: = 0.1.92 changed to = 0.1.93
Are you sure you want to continue? (Y/N) N
You said no, so I'm done here.
Would you like to reset your local development.json to match the server?? (Y/N) Y
<snip>
development.json reset.
Spork: Features
Spork: Features• Default Environments
Spork: Features• Default Environments
• Git support
Spork: Features• Default Environments
• Git support
• Chat notifications + Gist (upload and promote)
Spork: Features• Default Environments
• Git support
• Chat notifications + Gist (upload and promote)
• IRCCat
Spork: Features• Default Environments
• Git support
• Chat notifications + Gist (upload and promote)
• IRCCat
• Hipchat (courtesy of Secondmarket)
Features: Chat Notifications
[19:43:46] <irccat> CHEF: pmcdonnell uploaded and froze cookbook immount version 0.0.24
[19:44:00] <irccat> CHEF: pmcdonnell uploaded environment production https://github.etsycorp.com/gist/385043
[19:44:01] <irccat> CHEF: pmcdonnell uploaded environment development https://github.etsycorp.com/gist/385044
Spork: Features• Default Environments
• Git support
• Chat notifications + Gist (upload and promote)
• IRCCat
• Hipchat (courtesy of Secondmarket)
• Graphite (on promote)
Features: Graphite
Spork: Features• Default Environments
• Git support
• Chat notifications + Gist (upload and promote)
• IRCCat
• Hipchat (courtesy of Secondmarket)
• Graphite (on promote)
• Foodcritic (on upload)
Features: Foodcritic
$> knife spork upload system
<snip>
Lint checking system...
ERROR: Lint check failed. Halting upload.
ERROR: Lint check output:
ERROR: ETSY003: Execute resource used to run curl or wget commands: /Users/jcowie/dev/etsy/chef/cookbooks/system/recipes/dev-ssl.rb:41
Use the Source!
tiny.cc/velocity2012
We’re hiring!
BoF on Tuesdayor just come and say Hi :)
Top Related