Oscon2011 tutorial

OSCON 2011: Getting Started with Chef

Joshua Timbermanjoshua@opscode.com, @jtimberman

Aaron Petersonaaron@opscode.com, @metaxis

http://opscode.com

Monday, July 25, 2011

Meta Information

• OSCON tutorials are recorded

• Rate the tutorial and comment

• http://bit.ly/chef-oscon2011

• Twitter:

• #oscon

• @opscode, #opschef

• @jtimberman, @metaxis

• Slides and Code will be posted

Who are we?

• Joshua Timberman

• Aaron Peterson

Who are you?

• System administrators?

• Developers?

• “Business” People?

http://www.flickr.com/photos/timyates/2854357446/sizes/l/

Agenda

• Tutorial Logistics

• Hows and whys

• Getting Started

• Anatomy of a Chef Run

• Hands on configuring a node

• Common patterns & best practices

• Question/Answer

http://www.flickr.com/photos/koalazymonkey/3590953001/

What are we talking about here?

http://www.flickr.com/photos/peterkaminski/2174679908/

Managing Infrastructure is Hard

• Sysadmins:

• Setup production machines

• Manage deployed application(s)

System administrators...

• Install packages

• Configure running services

• OS settings

• User management

• Monitoring and trending integration

Managing Infrastructure is Hard

• Developers:

• Setup local machine

• Deploy application for testing

Developers...

• Developers want self-service

• Full application stack

• Abstract the details

Automation is Good

• Operable

• Reasonable

• Flexible

• Repeatable

The Chef Framework

• Reasonability

• Flexibility

• Library & Primitives

• TIMTOWTDI

The Chef Tool(s)

• ohai - information gathering

• chef-client - configuration agent

• knife - command-line API tool

• shef - console debugger

The Chef API

• RSA key authentication w/ Signed Headers

• RESTful API w/ JSON

• Search Service

• Derivative Services

The Chef Community

• Apache License, Version 2.0

• 360+ Individual contributors

• 70+ Corporate contributors

• Dell, Rackspace,VMware, RightScale, Basho Technologies, and more

• http://community.opscode.com

• 260+ cookbooks

Getting Started with Chef

git clone git://github.com/opscode/oscon2011-chef-repo

Required Software

• SSH/SCP

• Git

• Build toolchain (gcc and friends)

• Ruby (1.8.7 or 1.9.2)

• RubyGems (1.3.7+)

• Chef (0.10.0+)

Why Opscode Hosted Chef?

• Limited time for tutorial

• Free up to 5 nodes

• Chef Server API

• Open Source Chef Server

Source Code Repository

• Chef Repository for OSCON 2011

• git clone git://github.com/opscode/oscon2011-chef-repo

Files from Opscode Hosted Chef Signup

• Knife configuration

• .chef/knife.rb

• User certificate

• .chef/USER.pem

• Validation certificate

• .chef/ORGNAME-validator.pem

Verify Access

% knife client listoscon2011-validator

% knife node from file dummy.example.com.json Updated Node dummy.example.com!

% knife node listdummy.example.com

% knife node show dummy.example.comNode Name: dummy.example.comEnvironment: _defaultFQDN: dummy.example.comIP: 10.1.1.1Run List: Roles: Recipes Platform: centos 5.5

Virtual Machine Setup

• Setup outside scope of this tutorial

• Linux Virtual Machine or Cloud Instance

• SSH access as root or user w/ sudo

A quick tour of Chef

Chef runs on your systems

API Clients authenticate to the Chef Server

Each system running Chef is a managed Node

Nodes have attributes and a list of things to run

Roles are a description of what a node should be

Chef configures Resources on your systems

Recipes are lists of resources

Cookbooks are packages for Recipes and related files

Let’s manage some infrastructure...

Managing Infrastructure

• Write or download cookbooks

• Create a role that uses the cookbooks

• Deploy cookbooks and role to Chef Server

• Apply the role to a node

• Run Chef on the node

Anatomy of a Chef Run

Profile the Node with Ohai

Run Ohai

• Run `ohai | less` on your system.

• Marvel at the amount of data it returns.

Authenticate

Retrieve Node from Chef Server

Sync Cookbooks from Chef Server

Load Cookbooks

Load Recipes

Converge

Save Node to Chef Server

Break Time

• Questions from 1st half

• Hands on in 2nd half

http://www.flickr.com/photos/refractedmoments/65794219/

Questions?

http://www.flickr.com/photos/oberazzi/318947873/

Reasoning about Infrastructure

• Break down complexity into components you can think about.

• Think about commonality and differences between systems and applications.

• Capture these in roles.

• For a given application, think about requirements to fulfill its job.

• Think about how to meet the requirements.

Concrete use case

• Stand in for common patterns

• Things we want on all systems in the infrastructure.

• User management

• Essential network service (NTP)

Upload Chef Repository

% knife role from file base.rb

% knife cookbook upload -a

% knife data bag create users

% knife data bag from file users luke.json

% knife data bag from file users leia.json

Configure a node

• Invoke action from the local workstation to happen on a remote machine over SSH.

• Virtual Machine IP address

• SSH key or password for root/privileged (sudo) user

• Optional: Use a cloud computing provider (See README.md)

Knife Bootstrap

knife bootstrap FQDN (options) -d DISTRO Target a specific distro (default ubuntu) -i IDENTITY_FILE SSH identity file for authentication -r RUN_LIST Run list for the node -P PASSWORD The ssh password -x USERNAME The ssh username (default root) --sudo Execute bootstrap with sudo

% knife bootstrap --help

% knife help bootstrap # full man page!

Bootstrap Cloud Instances

• Knife works with Cloud providers through plugins

• Knife Cloud plugins use Fog

• Cloud instances are launched via their API then provisioned with bootstrap

• Additional RubyGems

• knife-ec2, knife-rackspace, etc

• Additional Knife Configuration

Configure a node

# Append -Ppassword or -i ~/.ssh/ssh-private-key-for-you to ssh# Ubuntu:knife bootstrap $IPADDRESS -r 'role[base]'knife bootstrap $IPADDRESS -r 'role[base]' -x ubuntu --sudo

# Debian 6:knife bootstrap $IPADDRESS -r 'role[base]' -x rootknife bootstrap $IPADDRESS -r 'role[base]' -x username --sudo

# CentOS 5.x:knife bootstrap $IPADDRESS -r 'role[base]' -d centos5-gemsknife bootstrap $IPADDRESS -r 'role[base]' -d centos5-gems -x username --sudo

# Scientific Linux 6.x:knife bootstrap $IPADDRESS -r 'role[base]' -d scientific6-gemsknife bootstrap $IPADDRESS -r 'role[base]' -d scientific6-gems -x username --sudo

# Example (Ubuntu 10.04):knife bootstrap 172.16.156.130 -r 'role[base]' -x jtimberman --sudo -Poscon2011

What happened on the node?

recipe[ntp]

INFO: Processing package[ntp] action install (ntp::default line 27)INFO: package[ntp] installed version 1:4.2.4p8+dfsg-1ubuntu2.1INFO: Processing package[ntp] action install (ntp::default line 27)INFO: package[ntp] installed version 1:4.2.4p8+dfsg-1ubuntu2.1INFO: Processing template[/etc/ntp.conf] action create (ntp::default line 31)INFO: template[/etc/ntp.conf] backed up to /var/chef/backup/etc/ntp.conf.chef-20110717131907INFO: template[/etc/ntp.conf] mode changed to 644INFO: template[/etc/ntp.conf] updated contentINFO: Processing service[ntp] action enable (ntp::default line 39)INFO: Processing service[ntp] action start (ntp::default line 39)

[ ... end of run (delayed) ... ]INFO: template[/etc/ntp.conf] sending restart action to service[ntp] (delayed)INFO: Processing service[ntp] action restart (ntp::default line 39)INFO: service[ntp] restarted

SSH to the Node and inspect

% ssh 172.16.156.130

% dpkg -l ntpii ntp 1:4.2.4p8+dfsg Network Time Protocol daemon and

% grep server /etc/ntp.conf server 0.pool.ntp.org server 1.pool.ntp.org

% /etc/init.d/ntp status * NTP server is running

% ls /etc/rc2.d/*ntp/etc/rc2.d/S23ntp

recipe[users::sysadmin]

INFO: Processing user[luke] action create (users::sysadmins line 41)INFO: user[luke] createdINFO: Processing directory[/home/luke/.ssh] action create (users::sysadmins line 51)INFO: directory[/home/luke/.ssh] created directory /home/luke/.sshINFO: directory[/home/luke/.ssh] owner changed to 2001INFO: directory[/home/luke/.ssh] group changed to 2001INFO: directory[/home/luke/.ssh] mode changed to 700INFO: Processing template[/home/luke/.ssh/authorized_keys] action create (users::sysadmins line 57)INFO: template[/home/luke/.ssh/authorized_keys] owner changed to 2001INFO: template[/home/luke/.ssh/authorized_keys] owner changed to 2001INFO: template[/home/luke/.ssh/authorized_keys] updated content

INFO: Processing user[leia] action create (users::sysadmins line 41)INFO: user[leia] createdINFO: Processing directory[/home/leia/.ssh] action create (users::sysadmins line 51)INFO: directory[/home/leia/.ssh] created directory /home/leia/.sshINFO: directory[/home/leia/.ssh] owner changed to 2002INFO: directory[/home/leia/.ssh] group changed to 2002INFO: directory[/home/leia/.ssh] mode changed to 700INFO: Processing template[/home/leia/.ssh/authorized_keys] action create (users::sysadmins line 57)INFO: template[/home/leia/.ssh/authorized_keys] owner changed to 2002INFO: template[/home/leia/.ssh/authorized_keys] owner changed to 2002INFO: template[/home/leia/.ssh/authorized_keys] updated contentINFO: Processing group[sysadmin] action create (users::sysadmins line 66)INFO: group[sysadmin] created

recipe[users::sysadmins]

% ssh 172.16.156.130

% getent passwd luke leialuke:x:2001:2001:Force is strong with this one:/home/luke:/bin/bashleia:x:2002:2002:There is another:/home/leia:/bin/bash

# ls ~{luke,leia}/.ssh/home/luke/.ssh:authorized_keys

/home/leia/.ssh:authorized_keys

recipe[sudo]

INFO: Processing package[sudo] action upgrade (sudo::default line 20)INFO: Processing template[/etc/sudoers] action create (sudo::default line 24)INFO: template[/etc/sudoers] backed up to /var/chef/backup/etc/sudoers.chef-20110717131908INFO: template[/etc/sudoers] mode changed to 440INFO: template[/etc/sudoers] updated content

recipe[sudo]

# grep ALL /etc/sudoersroot ALL=(ALL) ALL%sysadmin ALL=(ALL) ALL

What happened on the Chef Server?

Chef Repository on Chef Server

% knife role listbase

% knife cookbook listntp 1.0.0sudo 1.0.0users 1.0.0

% knife data bag listusers

% knife data bag show users leia luke

Base Role

% knife role show basechef_type: roledefault_attributes: {}description: Base role applied to all systemsenv_run_lists: {}json_class: Chef::Rolename: baseoverride_attributes: {}run_list: recipe[ntp], recipe[users::sysadmins], recipe[sudo]

NTP Cookbook

package "ntp" do action :installend

template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, "service[ntp]"end

service "ntp" do action [:enable, :start]end

NTP configuration (template)

template "/etc/ntp.conf" do source "ntp.conf.erb" owner "root" group "root" mode 0644 notifies :restart, "service[ntp]"end

Template source:<% node[:ntp][:servers].each do |ntpserver| -%> server <%= ntpserver %><% end -%><% end -%>

Cookbook Attributes:default[:ntp][:servers] = ["0.pool.ntp.org", "1.pool.ntp.org"]

NTP service management

template "/etc/ntp.conf" do # ... notifies :restart, "service[ntp]"end

service "ntp" do action [:enable, :start]end

Sysadmin users data bag items

% cat data_bags/users/luke.json{ "id": "luke", "ssh_keys": "ssh-rsa For example purposes only", "groups": "sysadmin", "uid": 2001, "shell": "/bin/bash", "comment": "Force is strong with this one"}

% cat data_bags/users/leia.json{ "id": "leia", "ssh_keys": "ssh-rsa For example purposes only", "groups": "sysadmin", "uid": 2002, "shell": "/bin/bash", "comment": "There is another"}

users::sysadmins recipe

search(:users, 'groups:sysadmin') do |u|

user u['id'] do uid u['uid'] gid u['id'] shell u['shell'] comment u['comment'] supports :manage_home => true home "/home/#{u['uid']}" end

directory "#{home_dir}/.ssh" do owner u['id'] group u['id'] mode "0700" end

template "#{home_dir}/.ssh/authorized_keys" do source "authorized_keys.erb" owner u['id'] group u['id'] mode "0600" variables :ssh_keys => u['ssh_keys'] endend

Sudo cookbook

package "sudo" do action :upgradeend

template "/etc/sudoers" do source "sudoers.erb" mode 0440 owner "root" group "root" variables( :sudoers_groups => node['authorization']['sudo']['groups'], :sudoers_users => node['authorization']['sudo']['users'], :passwordless => node['authorization']['sudo']['passwordless'] )end

Sudoers template

Template source:

root ALL=(ALL) ALL%sysadmin ALL=(ALL) <%= "NOPASSWD:" if @passwordless %>ALL

Cookbook attributes:

default['authorization']['sudo']['passwordless'] = false

Rendered content:

root ALL=(ALL) ALL%sysadmin ALL=(ALL) ALL

% knife node list dummy.example.com ubuntu1004test.example.com

% knife node show ubuntu1004test.example.comNode Name: ubuntu1004test.example.comEnvironment: _defaultFQDN: ubuntu1004test.example.comIP: 172.16.156.130Run List: role[base]Roles: baseRecipes ntp, users::sysadmins, sudoPlatform: ubuntu 10.04

% knife node show --help% knife help node

Searching the Server

# Search nodes:% knife search node "role:base"% knife search node "platform:ubuntu"% knife search node "platform:centos"

# Search roles:% knife search role "run_list:recipe\[users*"

# Search data bags (bag name is the index):% knife search users "groups:sysadmin"% knife search users "shell:*bash"

Common Patternsand

Best Practices

Common Patterns

• Install a package

• Update a configuration file

• Restart a service

Common Patterns

• Search for nodes with a particular role

• Search for data bag items

• Make decisions or render templates based on search results.

Search example in a recipe

pool_members = search("node", "role:webserver")

template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end

Common Patterns

• Ask questions about the infrastructure.

• Target a subset of servers and take action.

• Search with Roles

• Search with Node Attributes

• Parallel execution of commands.

Operational Use Case

% knife ssh platform:ubuntu 'vmstat'xwing.example.com procs -----------memory---------- ...xwing.example.com r b swpd free buff cache ...xwing.example.com 0 0 0 684804 461656 6052916 ...tiefighter.example.com procs -----------memory---------- ...tiefighter.example.com r b swpd free buff cache ...tiefighter.example.com 0 0 0 169020 708844 6120008 ...

Best Practices: Cookbooks

• Publicly shared cookbooks:

• http://community.opscode.com

• Create your own

• knife cookbook create foo

• $EDITOR cookbooks/foo/recipes/default.rb

Getting Community Cookbooks

# Install apache2 cookbook from site in Git chef-repo% knife cookbook site install apache2

# Download and install apache2 cookbook in non-Git chef-repo% knife cookbook site download apache2

% tar -zxf apache2-VERSION.tar.gz -C cookbooks

Best Practices: Cookbooks

• Cookbook for each service

• Recipe for each component or deployment of the service

• Set sane defaults in attributes files

• Modify attributes through roles for specific usage requirements

Best Practices: Roles

• Roles are descriptions

• webserver

• database_master

• load_balancer

• Set role-specific attributes when necessary

• listen ports, deploy locations, etc

Best Practices: Nodes

• Use “Just Enough OS”

• Use fully updated systems

• Kickstart, AMI, etc

• Ensure system clock is synchronized

• Be ready to deploy from scratch

Managing Resources

• Chef’s primary purpose is managing resources on nodes.

• Think in terms of resources vs commands

• Chef comes with 28 kinds of resources

• You can create your own resources in cookbooks

Thinking in terms of resources

• package vs yum install

• service vs chkconfig

• template vs echo ‘coolstuff’ >> /etc/config

• or sed ‘s/badstuff/coolstuff/’...

• mode, owner and group parameters vs chmod/chown

• http://wiki.opscode.com/display/chef/Resources

FAQ: Chef vs [Other Tool]

http://www.flickr.com/photos/gesika22/4458155541/

FAQ: How do you test recipes?

FAQ: Testing

• You launch cloud instances and watch them converge.

• You use Vagrant with a Chef Provisioner

FAQ: Testing

• You buy Stephen Nelson-Smith’s book!

FAQ: How does Chef scale?

FAQ: Scale

• The Chef Server is a publishing system.

• Nodes do the heavy lifting.

• Chef scales like a service-oriented web application.

• Opscode Hosted Chef was designed and built for massive scale.

http://www.flickr.com/photos/amagill/61205408/

Questions?

http://www.flickr.com/photos/oberazzi/318947873/

• http://bit.ly/chef-oscon2011

• http://opscode.com

• @opscode, #opschef

• irc.freenode.net, #chef, #chef-hacking

• http://lists.opscode.com

Thanks!

http://opscode.com@opscode#opschef

Oscon2011 tutorial

Technology

Transcript of Oscon2011 tutorial

(Tutorial) Photo Editing Tutorial (Glamour Look)

Tutorial Mikrotik Tutorial Mk Doc

ENVI Tutorial: Vegetation Analysis - SELabspweather.com/enviuser/tutorial/misc/Vegetation_Analysis_Toolkit.pdf · Tutorial: Vegetation Analysis 2 ENVI Tutorial: Vegetation Analysis

Tutorial AAAI 2006 Tutorial Forum

IMS tutorial: IMS tutorial:

Tutorial Crocodile Technology - Tutorial Crocodile Technology

Python Tutorial - edisciplinas.usp.br filePython Tutorial

IBM DOORS Tutorial Systems Engineering Tutorial

Wirecast 8.3 Tutorial for Mac - Telestream · Tutorial Wirecast 8.3 Tutorial for Mac Wirecast Tutorial | 273942 February 2018

Photoshop Tutorial: Cinematic Effect Tutorial

Tutorial Letter 201/1/2018The study guide The text of this module’s tutorial matter Tutorial Letter 101 The general information tutorial letter Tutorial Letter 201 This tutorial

SE4 Tutorial Tutorial 7

Tutorial 8: Mass modeling - Esri Supportdownloads.esri.com/resources/cityengine/tutorial_8_mass_modeling.… · Tutorial 8: Mass modeling Download items • Tutorial data • Tutorial

StarUML Tutorial - OpenS_ StarUML-Tutorial Paag1

Tutorial 1 - iLogic Basic Tutorial

Support.ebsco.com Tutorial de Mi EBSCOhost Tutorial.

GIVING A TUTORIAL ACADEMIC ENGLISH II. TUTORIAL DEVELOPMENT You will learn how to: Plan a tutorial Prepare a tutorial Practice a tutorial Present a tutorial.

Python Tutorial Ets Tutorial

Support.ebsco.com Tutorial do Meu EBSCOhost Tutorial.

Tutorial - TUIASIchar.tuiasi.ro/doace/ · Jtest Tutorial 1 Tutorial TutorialJtest Tutorial Welcome to the Jtest Tutorial. ... ParaSoft’s Jcontract to check Design by Contract contracts