Chef - Configuration Management for the Cloud

Configuration Management in the Cloud with Chef

PDX Cloud

James [email protected]: jamesc_000

GitHub: jamescwww.opscode.com

Tuesday, April 30, 13

mailto:[email protected]


http://www.opscode.com


• Instant infrastructure

• Unlimited capacity

• Autoscaling

• No commitment

• Immediate replacement

Clouds are great !


APIs are awesome!

• You can provision compute resources in seconds

• You can provision storage resources in seconds

• You can provision network resources in seconds

• That’s cool.

http://www.flickr.com/photos/jdhancock/3634246981/




The Dark Side of the Cloud


• Performance

• Security

• Price

• Reliability

Why not the Cloud?


See Node

Application Server


See Nodes

Application Server

Application Database


See Nodes Grow

Application Server

Application Databases


Application Servers


See Nodes Grow


Application Servers


Load Balancer

See Nodes Grow


See Nodes Grow

Application Servers


Load Balancers


See Nodes Grow

Application Servers

Application Database Cache

Load Balancers



Tied together with Config

Application Servers


Load Balancers



Infrastructure is a Snowflake

Application Servers


Load Balancers

Floating IP?



Evolving Complexity

Load Balancers

Application Servers

NoSQL

Database Slaves

ApplicationCache

Database Cache

Database


Complexity Grows Quickly

DC1

DC3

DC2


http://www.flickr.com/photos/16339684@N00/2681435235/

And it Continues to Evolve


http://www.flickr.com/photos/16339684@N00/2681435235/

And it Continues to Evolve

Ok, so I’ve got a problem.

What’s the solution ?


Golden Images are not the answer

• Gold is heavy

• Hard to transport

• Hard to mold

• Easy to lose configuration detail

http://www.flickr.com/photos/garysoup/2977173063/




Configuration Management and Automated Systems Integration

is the Answer

http://www.flickr.com/photos/philliecasablanca/3354734116/Tuesday, April 30, 13

http://www.flickr.com/photos/philliecasablanca/3354734116/

http://www.flickr.com/photos/philliecasablanca/3354734116/

• Turn code and hardware into infrastructure

• From bare metal to services in production

• Scale applications as needed

• Conform to policy

• Align to business goals

Configuration Management ?


Chef - Infrastructure as Code

http://www.flickr.com/photos/louisb/4555295187/

• Programmatically provision and configure

• Treat like any other code base

• Reconstruct business from code repository, data backup, and bare metal resources.




• Chef-Client generates configurations directly on nodes from their run list

• Reduce management complexity through abstraction

• Store the configuration of your programs in version control

http://www.flickr.com/photos/ssoosay/5126146763/

Nodes




Collections of Resources

• Networking

• Files

• Directories

• Symlinks

• Mounts

• Routes

• Users

• Groups

• Tasks

• Packages

• Software

• Services

• Configurations

• Other Stuffhttp://www.flickr.com/photos/stevekeys/3123167585/


http://www.flickr.com/photos/stevekeys/3123167585/

http://www.flickr.com/photos/stevekeys/3123167585/

Declarative Interface to Resources

• Define policy

• Say what, not how

• Pull not Push

http://www.flickr.com/photos/bixentro/2591838509/Tuesday, April 30, 13

http://www.flickr.com/photos/bixentro/2591838509/

http://www.flickr.com/photos/bixentro/2591838509/

Ruby!

extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } endextra_packages.each do |pkg| package pkg do action :install endend


Or thissearch(:users, '*:*') do |u| user u['id'] do uid u['uid'] shell u['shell'] home "/home/#{u['id']}" end directory "#{home_dir}/.ssh" do

owner u['id'] group u['gid'] mode "0700" end template "#{home_dir}/.ssh/authorized_keys" do source "authorized_keys.erb" owner u['id'] group u['id'] mode "0600" variables :ssh_keys => u['ssh_keys']

endend


Recipes and Cookbooks

• Recipes are collections of Resources

• Cookbooks contain recipes, templates, files, custom resources, etc

• Code re-use and modularity

• Hundreds already on Community.opscode.com

http://www.flickr.com/photos/shutterhacks/4474421855/Tuesday, April 30, 13

http://www.flickr.com/photos/shutterhacks/4474421855/

http://www.flickr.com/photos/shutterhacks/4474421855/

http://www.flickr.com/photos/kathycsus/2686772625

• IP addresses

• Hostnames

• FQDNs

• Search for nodes with Roles

• Find configuration data

Search




pool_members = search("node","role:webserver”)

template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end

Pass Results to Templates


# Set up application listeners here.listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>

Pass Results to Templates


Jboss App

Memcache

Postgres Slaves

Postgres Master

So when this

NagiosGraphite


Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

Becomes this


Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

Updates can be automatic


NagiosGraphite

Count the resources

Jboss App

Memcache

Postgres Slaves

• Load balancer config

• Nagios host ping

• Nagios host ssh

• Nagios host HTTP

• Nagios host app health

• Graphite CPU

• Graphite Memory

• Graphite Disk

• Graphite SNMP

• Memcache firewall

• Postgres firewall

• Postgres authZ config

• 12+ resource changes for 1 node addition


http://www.flickr.com/photos/evelynishere/2798236471/

CLONING CANNOT COPE WITH THIS

• Chef can.




Build anything

• Simple internal applications

• Complex external applications

• Workstations

• Hadoop clusters

• IaaS infrastructure

• PaaS infrastructure

• SaaS applications

• Storage systems

• You name it

http://www.flickr.com/photos/hyku/245010680/




And manage it simply

http://www.flickr.com/photos/helico/404640681/

• Automatically reconfigure everything

• Linux, Windows, Unixes, BSDs

• Load balancers

• Metrics collection systems

• Monitoring systems

• Cloud migrations become trivial




knife


Upload your infrastructureknife cookbook upload apt

knife cookbook upload chef-client

knife cookbook upload java

knife cookbook upload jpackage

knife cookbook upload ntp

knife cookbook upload sudo

knife cookbook upload tomcat

knife cookbook upload users

knife cookbook upload sample

knife role from file base-cloud.rb

knife role from file tc.rb

knife role from file sample.rb

knife data bag create users

knife data bag from file users jamesc.jsonTuesday, April 30, 13

Build it somewhere

#EC2knife ec2 server create -S jamesc -i ~/.ssh/jamesc.pem -x ubuntu -G default -I ami-a7a97dce -f m1.small -d omnibus -r 'role[base-cloud],role[tc],role[sample]'

#Rackspaceknife rackspace server create --image 110 --flavor 2 -i ~/.ssh/jamesc.pem -d omnibus -r 'role[base-cloud],role[tc],role[sample]'

#CloudStackknife cs server create -S "small instance" -T "CentOS 5.5(64-bit) no GUI (KVM)" -i ~/.ssh/jamesc.pem -d omnibus -r 'role[base-cloud],role[tc],role[sample]'

#Ubuntu Linuxknife bootstrap test.lab -r 'role[webserver]' -i ~/.ssh/jamesc.pem -x ubuntu --sudo -d omnibus -r 'role[base-cloud],role[tc],role[sample]'


knife ec2

$ knife ec2Available ec2 subcommands: (for details, knife SUB-COMMAND --help)

** EC2 COMMANDS **knife ec2 flavor list (options)knife ec2 instance data (options)knife ec2 server create (options)knife ec2 server delete SERVER [SERVER] (options)knife ec2 server list (options)

$ knife ec2 server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I ami-4721882e -f m1.small -r 'role[webserver]'


knife openstack

$ knife openstackAvailable openstack subcommands: (for details, knife SUB-COMMAND --help)

** OPENSTACK COMMANDS **knife openstack flavor list (options)knife openstack image list (options)knife openstack server create (options)knife openstack server delete SERVER [SERVER] (options)knife openstack server list (options)

$ knife openstack server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I 1231 -f standard.small -r 'role[webserver]'


Chef for Infrastructure Portability

• knife ec2

• knife rackspace

• knife hp

• knife google

• knife azure

• knife cloudstack

• knife openstack

• knife vsphere

• ... and many others


The Chef Community

• Apache License, Version 2.0

• 850+ Individual contributors

• 150+ Corporate contributors

• HP, Dell, Rackspace, VMware, Joyent, Calxeda, Heroku, SUSE and many more

• 550+ cookbooks

• http://community.opscode.com


http://community.opscode.com

http://community.opscode.com

Summary

• Every infrastructure is a unique snowflake

• You need tools to let you do what you want

• You need the power to grow your infrastructure

• You need the ability to change your cloud provider

• Automated Configuration Management is the solution


Questions?

http://www.flickr.com/photos/mrchippy/443960682/

Questions?




Thanks!

James [email protected]: jamesc_000

Github: jamescwww.opscode.com






Chef - Configuration Management for the Cloud

Technology

Transcript of Chef - Configuration Management for the Cloud