Chef: The Swiss Army Knife of Cloud InfrastructureMatt Ray SCALE Build a Cloud Day February 21, 2014

Introductions• Matt Ray

• Director of Cloud Integrations at Chef

• matt@getchef.com

• mattray GitHub|IRC|Twitter

We have a problem…

http://www.flickr.com/photos/michaelheiss/3090102907/

Complexity

Items of Manipulation (Resources)• Nodes • Networking • Files • Directories • Symlinks • Mounts

• Routes • Users • Groups • Packages • Services • Filesystems

Application

A tale of growth...

Application

Application Database

Add a database

Application

App Databases

Make database redundant

App Servers

App Databases

Application server redundancy

App LB

App Servers

App Databases

Add a load balancer

App LBs

App Servers

App Databases

Webscale!

App LBs

App Servers

App DB Cache

App DBs

Now we need a caching layer

App LBs

App Servers

App DB Cache

App DBs

Infrastructure has a Topology

Round Robin DNS

App Servers

App DB Cache

App DBs

Floating IP?

Your Infrastructure is a Snowflake

App LBs

App Servers

< Shiny!

DB slaves

Cache

DB Cache

DBs

Complexity Increases Quickly

Are we monitoring??

™

The Chef Framework• Reasonability • Flexibility • Library & Primitives • TIMTOWTDI • Sane defaults

http://www.flickr.com/photos/wonderlane/3609342683/sizes/l/in/photostream/

The Chef Tool(s)• ohai • chef-client • chef-shell • knife • The Ruby language

Omnibus - Full Stack Native Packages

The Chef API• HTTPS, RESTful API w/ JSON, RSA key auth • Infrastructure data store such as node data • Search Service • Derivative Services?

http://www.flickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/

The Chef Community• Apache License, Version 2.0 • Thousands of Individual and Corporate contributors. • Thousands of cookbooks available from the

community • http://community.opscode.com

How does it work?

http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg

Chef is Infrastructure as Code• Programmatically

provision and configure

• Treat like any other code base

• Reconstruct business from code repository, data backup, and bare metal resources.http://www.flickr.com/photos/louisb/4555295187/

Programs• Chef generates

configurations directly on nodes from their run list

• Reduce management complexity through abstraction

• Store the configuration of your programs in version control

http://www.flickr.com/photos/ssoosay/5126146763/

Declarative Interface to Resources• Define Policy • Say what, not how • Pull not Push

http://www.flickr.com/photos/bixentro/2591838509/

That Looks Like Thispackage "apache2"

template "/etc/apache2/apache2.conf" do! source "apache2.conf.erb"! owner "root"! group "root"! mode "0644"! variables(:allow_override => "All")! notifies :reload, "service[apache2]"!end

service "apache2" do! action [:enable,:start]! supports :reload => true!end

http://www.flickr.com/photos/peterrosbjerg/3913766224/

Chef Nodes• Chef runs on nodes • Chef nodes do the heavy lifting • Authority about themselves • Stored on the server when using Chef Server • Indexed for search

Search• Search for nodes with Roles • Find Topology Data !

• IP addresses • Hostnames • FQDNs

http://www.flickr.com/photos/kathycsus/2686772625

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

So when this...

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

...becomes this

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

!!! !!

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

...this can happen automatically

NagiosGraphite

!!! !!

NagiosGraphite

Memcache

Postgres Slaves

• Load balancer config

• Nagios host ping

• Nagios host ssh

• Nagios host HTTP

• Nagios host app health

• Graphite CPU

• Graphite Memory

• Graphite Disk

• Graphite SNMP

• Memcache firewall

• Postgres firewall

• Postgres authZ config

•12+ resource changes for 1 node addition

Count the Resources

Jboss App

Landscape of Chef-managed Infrastructure

knife

knife - with the Chef Server• knife node

• create / edit / delete • list

• knife cookbook ... • knife role ... • knife environment ...

• SSH to the machine given existing credentials • Install the Chef Client • Register with the Chef Server • Run the initial Run List • Now managed with Chef!

$ knife bootstrap SERVER -r ‘role[webserver]’

knife bootstrap

gem install knife-cloudstack

Chef & CloudStack

• ** CS COMMANDS ** • knife cs aag list (options) • knife cs account list (options) • knife cs cluster list (options) • knife cs config list (options) • knife cs disk list (options) • knife cs domain list (options) • knife cs firewallrule create hostname 8080:8090:TCP:10.0.0.0/24 • knife cs firewallrule list (options) • knife cs forwardrule create hostname 8080:8090:TCP • knife cs host list (options)

knife cs

knife-cloudstack 1/4

• knife cs hosts • knife cs iso list (options) • knife cs keypair create KEY_NAME (options) • knife cs keypair delete KEY_NAME (options) • knife cs keypair list (options) • knife cs network list (options) • knife cs oscategory list (options) • knife cs ostype list (options) • knife cs pod list (options) • knife cs project list (options)

knife cs

knife-cloudstack 2/4

• knife cs publicip list (options) • knife cs router list (options) • knife cs securitygroup list (options) • knife cs server add nic SERVERID NETWORKID (--ipaddress X.X.X.X) • knife cs server create [SERVER_NAME] (options) • knife cs server delete SERVER_NAME [SERVER_NAME ...] (options) • knife cs server list (options) • knife cs server reboot SERVER_NAME [SERVER_NAME ...] (options) • knife cs server remove nic SERVERID NICID • knife cs server start SERVER_NAME [SERVER_NAME ...] (options)

knife cs

knife-cloudstack 3/4

• knife cs server stop SERVER_NAME [SERVER_NAME ...] (options) • knife cs service list (options) • knife cs stack create JSON_FILE (options) • knife cs stack delete JSON_FILE (options) • knife cs template create NAME (options) • knife cs template extract NAME (options) • knife cs template list (options) • knife cs template register NAME (options) • knife cs user list (options) • knife cs volume create NAME (options) • knife cs volume list (options) • knife cs zone list (options)

knife cs

knife-cloudstack 4/4

• Instant infrastructure • Unlimited capacity • Autoscaling • No commitment • Immediate replacement

Why the Cloud?

• Reliability • Performance • Security • Price

Why not the Cloud?

Data Gravity

Know our escape plan for every infrastructure provider

Chef for Infrastructure Portability

• knife rackspace

• knife openstack

• knife azure

• knife cloudstack

• knife ec2

• knife google

• knife hp

• knife vsphere

• ...and many others

• AWS

• Rackspace

• HP

• Google

• Azure

• many others

™

Desktop, Virtualization & Cloud

• Vagrant

• VMware

• CloudStack

• Eucalyptus

• OpenStack

• bare metal

• AWS

• Rackspace

• HP

• Google

• Azure

• many others

Desktop, Virtualization & Cloud

• Vagrant

• VMware

• CloudStack

• Eucalyptus

• OpenStack

• bare metal

™

What does this all mean?•Every infrastructure is a unique snowflake •Understand the costs associated with the features of your platform(s) of choice. •Chef enables Infrastructure Portability •Use the same infrastructure code for wherever you deploy •"Data Gravity" is a concern

™

There’s a lot more...• Attributes • Environments • Roles • Chef Handlers • LWRPs • Workflow Helpers • Testing Frameworks • and even more awesome

Thanks!Matt Ray matt@getchef.com @mattray