Languages
Pages
Legal
Achieving Infrastructure Portability with Chef
CloudOpen 2012
Matt [email protected]
IRC/Twitter/GitHub: mattraywww.opscode.com
Data Centers...
To the Cloud!
• Instant infrastructure
• Unlimited capacity
• Autoscaling
• No commitment
• Immediate replacement
Why the Cloud?
• Each cloud defines themselves against Amazon
• Entry into the market is easier
• Not a lot of price competition
• Feature parity is growing
Cloud Differentiation
The Dark Side of the Cloud
• Reliability
• Performance
• Security
• Price
Why not the Cloud?
Data Gravity
Know our escape plan for every infrastructure
provider
Why Chef?
See Node
Application Server
See Nodes
Application Server
Application Database
See Nodes Grow
Application Server
Application Databases
Application Servers
Application Databases
See Nodes Grow
Application Servers
Application Databases
Load Balancer
See Nodes Grow
See Nodes Grow
Application Servers
Application Databases
Load Balancers
See Nodes Grow
Application Servers
Application Database Cache
Load Balancers
Application Databases
Tied together with Config
Application Servers
Application Database Cache
Load Balancers
Application Databases
Infrastructure is a Snowflake
Application Servers
Application Database Cache
Load Balancers
Floating IP?
Application Databases
Evolving Complexity
Load Balancers
Application Servers
NoSQL
Database Slaves
ApplicationCache
Database Cache
Database
Complexity Grows Quickly
DC1
DC3
DC2
http://www.flickr.com/photos/16339684@N00/2681435235/
And it Continues to Evolve
That's great and all, but tell me about
Chef!
Chef is Infrastructure as Code
http://www.flickr.com/photos/louisb/4555295187/
• Programmatically provision and configure
• Treat like any other code base
• Reconstruct business from code repository, data backup, and bare metal resources.
• Chef-Client generates configurations directly on nodes from their run list
• Reduce management complexity through abstraction
• Store the configuration of your programs in version control
http://www.flickr.com/photos/ssoosay/5126146763/
Nodes
Collections of Resources
• Networking
• Files
• Directories
• Symlinks
• Mounts
• Routes
• Users
• Groups
• Tasks
• Packages
• Software
• Services
• Configurations
• Other Stuffhttp://www.flickr.com/photos/stevekeys/3123167585/
Declarative Interface to Resources
• Define policy
• Say what, not how
• Pull not Push
http://www.flickr.com/photos/bixentro/2591838509/
Ruby!
extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } endextra_packages.each do |pkg| package pkg do action :install endend
Recipes and Cookbooks
• Recipes are collections of Resources
• Cookbooks contain recipes, templates, files, custom resources, etc
• Code re-use and modularity
• Hundreds already on Community.opscode.com
http://www.flickr.com/photos/shutterhacks/4474421855/
http://www.flickr.com/photos/kathycsus/2686772625
• IP addresses
• Hostnames
• FQDNs
• Search for nodes with Roles
• Find configuration data
Search
pool_members = search("node","role:webserver”)
template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end
Pass Results to Templates
# Set up application listeners here.listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>
Pass Results to Templates
Jboss App
Memcache
Postgres Slaves
Postgres Master
So when this
NagiosGraphite
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
Becomes this
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
Updates can be automatic
NagiosGraphite
Count the resources
Jboss App
Memcache
Postgres Slaves
• Load balancer config
• Nagios host ping
• Nagios host ssh
• Nagios host HTTP
• Nagios host app health
• Graphite CPU
• Graphite Memory
• Graphite Disk
• Graphite SNMP
• Memcache firewall
• Postgres firewall
• Postgres authZ config
• 12+ resource changes for 1 node addition
Build anything
• Simple internal applications
• Complex external applications
• Workstations
• Hadoop clusters
• IaaS infrastructure
• PaaS infrastructure
• SaaS applications
• Storage systems
• You name it
http://www.flickr.com/photos/hyku/245010680/
And manage it simply
http://www.flickr.com/photos/helico/404640681/
• Automatically reconfigure everything
• Linux, Windows, Unixes, BSDs
• Load balancers
• Metrics collection systems
• Monitoring systems
• Cloud migrations become trivial
knife
knife with the Chef Server
• knife node
• create/delete/edit
• list
• knife cookbook ...
• knife role ...
• knife environment ...
knife bootstrap
knife bootstrap SERVER -r 'role[webserver]' -i ~/.ssh/id_rsa
• SSH to the machine given existing credentials
• Install the Chef Client
• Register with the Chef Server
• Run the initial Run List
• Now managed with Chef!
knife ec2
$ knife ec2Available ec2 subcommands: (for details, knife SUB-COMMAND --help)
** EC2 COMMANDS **knife ec2 flavor list (options)knife ec2 instance data (options)knife ec2 server create (options)knife ec2 server delete SERVER [SERVER] (options)knife ec2 server list (options)
$ knife ec2 server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I ami-4721882e -f m1.small -r 'role[webserver]'
knife openstack
$ knife openstackAvailable openstack subcommands: (for details, knife SUB-COMMAND --help)
** OPENSTACK COMMANDS **knife openstack flavor list (options)knife openstack image list (options)knife openstack server create (options)knife openstack server delete SERVER [SERVER] (options)knife openstack server list (options)
$ knife openstack server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I 1231 -f standard.small -r 'role[webserver]'
Chef for Infrastructure Portability
• knife ec2
• knife rackspace
• knife hp
• knife google
• knife azure
• knife cloudstack
• knife openstack
• knife vsphere
• ... and many others
The Chef Community
• Apache License, Version 2.0
• 850+ Individual contributors
• 150+ Corporate contributors
• HP, Dell, Rackspace, VMware, Joyent, Calxeda, Heroku, SUSE and many more
• 550+ cookbooks
• http://community.opscode.com
• Vagrant
• VMware
• CloudStack
• Eucalyptus
• OpenStack
• bare metal
• AWS
• Rackspace
• HP
• Azure
• many others
Desktop, Virtualization, Private & Public Clouds
• Vagrant
• VMware
• CloudStack
• Eucalyptus
• OpenStack
• bare metal
Desktop, Virtualization, Private & Public Clouds
• AWS
• Rackspace
• HP
• Azure
• many others
Tale of the Tape
• Artur Bergman, CEO at Fastly
• It's All About Speed
• http://youtu.be/qRnTejOMbZU
• Jason Stowe, CEO at Cycle Computing
• CycleCloud + Chef = 50,000-core Utility Supercomputer for Science
• http://youtu.be/cEaQB6e7G0Q
Chef and Abstractions
• Resources and Providers
• Cookbooks (may) normalize deployment
• Knife treats APIs the same
• Chef strives to not be opinionated
Environments
• Lock down versions of cookbooks
• stable releases vs. development
• Enforce attributes for deployment
• Ports, addresses, etc.
• Different run lists based on environment
• Debugging enabled?
Environments
• Use the same infrastructure code for wherever you deploy
• Development, QA, Pre-Production, Prod
• Role-based Access Controls to restrict the promotion of deployment code
TL;DL
• Every infrastructure is a unique snowflake
• Understand the costs associated with the features of your platform(s) of choice.
• Chef enables Infrastructure Portability
• "Data Gravity" is the primary concern
Top Related