Intro to Chef

Introduzione a Chef

Giacomo Bagnoli

Develer Workshops - 12 Settembre 2012

@gbagnoli Introduzione a Chef 12/09/2012 1 / 50

# whoami

• System Administrator

• Operations Engineer

• Python guy (having an affair with Ruby)

Outline

1. Introduction

2. Chef Overview

3. Example

Introduction

Infrastructure as code

Building and managing infrastructure programmatically

Enable the reconstruction of the business from:

• computing resources

• a source code repository

• data backups

Infrastructure as code (2)

source: Chef wiki

Provisioning Get new computing resources

Configuration Management Keeps track of all steps required to take baremetal resources to doing their job

System Integration Takes all configured systems and make them worktogether.

What is Chef

• A library/framework for configuration management

• A configuration management system

• A system integration platform

• An API for the infrastructure

• Open Source! (Apache License, version 2.0)

What is Chef

Opscode Chef

Chef Principles

Idempotent Describes states

Extensible Uses ruby as the DSL

Order It matters

Client-Server Thick Clients, thin Server

Various Flavours:

• Hosted on Opscode platform (chef-hosted)

• Open source Chef Server

• Serverless - chef-solo

• Private Chef (opscode-supported behind-the-firewall installation)

Chef Principles

Order It matters

Various Flavours:

Chef Principles

Order It matters

Various Flavours:

Chef Principles

Order It matters

Various Flavours:

Chef Principles

Order It matters

Various Flavours:

Chef Principles

Order It matters

Various Flavours:

Chef Principles

Order It matters

Various Flavours:

Chef Principles

Order It matters

Various Flavours:

Chef Principles

Order It matters

Various Flavours:

Chef Principles

Order It matters

Various Flavours:

Chef API

• A RESTful service with JSON responses

• RSA key authentication with Signed Headers

• Search Service

• Derivative (easy to integrate with other tools)

• . . . i.e pychef :-)

chef-client connects to the server consuming the API.CLI management tool knife and the webUI use the API too.

Chef API

• A RESTful service with JSON responses

• RSA key authentication with Signed Headers

• Search Service

• Derivative (easy to integrate with other tools)

• . . . i.e pychef :-)

chef-client connects to the server consuming the API.CLI management tool knife and the webUI use the API too.

Chef Architecture

RabbitMQ

CouchDBSolr

Chef API Server

Chef WebUI

ClientClient

Knife Knife

Indexer

Client

API Client

In chef, an API client provides the identity used to authenticaterequests to the API server.

The public half of the public/private of a key pair is stored in the db onthe server, while the private part is local to the client.

Each request to the API contains a request signature in the HTTP

headers.The request signature is computed by the hash of the request content andencrypted with the client private key, so it’s possible to verify the identityof the user/machine making the request.

API Client

A Node is a host that runs the chef-client.

• Has attributes

• Has a run list

• Has 0+ roles

• Belongs to an environment

In the common case, 1 host ⇔ 1 node ⇔ 1 client

run list

"run_list": {

"role[python_hosting]",

"recipe[postgresql::client]",

"recipe[chishop]"

• Have attributes

• Have a run list

• Declared in JSON or . . .

• Declared with the ruby DSL (automatically compiled to JSON)

If 1+ roles are in the node run list, the node run list is expanded

• Have attributes

• Have a run list

• Declared in JSON or . . .

• Declared with the ruby DSL (automatically compiled to JSON)

If 1+ roles are in the node run list, the node run list is expanded

Roles (2)

An example role (in ruby):

name "python_hosting"

description "Python App hosting"

default_attributes(

"nginx" => {

"default_site_enabled" => false

run_list(

"recipe[python::virtualenv]",

"recipe[uwsgi]",

"recipe[nginx]"

Attributes

Store node data (i.e. ip address, hostname, fqdn, database host address,etc.)There are four types of attributes (in order of precedence, lowest tohighest):

• default

• normal

• override

• automatic

Attributes can be set in:

• cookbooks

• environments

• roles

• nodes

Attributes

Store node data (i.e. ip address, hostname, fqdn, database host address,etc.)There are four types of attributes (in order of precedence, lowest tohighest):

• default

• normal

• override

• automatic

Attributes can be set in:

• cookbooks

• environments

• roles

• nodes

Attributes (2)

So, in the end, as attributes are deep-merged, the following precedenceapplies:

• default attributes applied in an cookbook

• default attributes applied in an environment

• default attributes applied in a role

• default attributes applied on a node directly in a recipe

• normal attributes applied in a cookbook

• normal attributes applied on a node directly in a recipe

• override attributes applied in an cookbook

• override attributes applied in an environment

• override attributes applied in a role

• override attributes applied on a node directly in a recipe

• automatic attributes generated by Ohai

Attributes (2)

So, in the end, as attributes are deep-merged, the following precedenceapplies:

• default attributes applied in an cookbook

• default attributes applied in an environment

• default attributes applied in a role

• default attributes applied on a node directly in a recipe

• normal attributes applied in a cookbook

• normal attributes applied on a node directly in a recipe

• override attributes applied in an cookbook

• override attributes applied in an environment

• override attributes applied in a role

• override attributes applied on a node directly in a recipe

• automatic attributes generated by Ohai

Attributes (3)

Automatic, ovverride and default are reset at the beginning of every run.Normal attributes persist between runs.

Attributes are searchable:

search(:node, ’platform:ubuntu’)

knife search node "platform:ubuntu"

Attributes (3)

Automatic, ovverride and default are reset at the beginning of every run.Normal attributes persist between runs.

Attributes are searchable:

search(:node, ’platform:ubuntu’)

knife search node "platform:ubuntu"

Attributes (4)

Summary:

• (sane) defaults in cookbooks

• . . . overridden in roles

• . . . and node-specific data as normal attributes on the node.

• override and node.set can be used to force values

Attributes (4)

Summary:

Attributes (4)

Summary:

Attributes (4)

Summary:

Attributes (4)

Summary:

Resources

Chef manages resources on a node.

Resources are specified in recipes, recipes stored in cookbooks.

The expanded run list specifies all the recipes (and thus the resources) tomanage on a given node.

Resources

Resources (2)

A resource

• has a type

• has a name

• has parameters

• takes actions

package "tar" do

version "1.16.1-1"

action :install

Actions are taken using providers, providers are chosen based on the nodeplatform.(i.e. the package resource installs packages using apt on debian/ubuntuand using yum on centos/RHEL)!

Resources (2)

A resource

• has a type

• has a name

• has parameters

• takes actions

package "tar" do

version "1.16.1-1"

action :install

Actions are taken using providers, providers are chosen based on the nodeplatform.(i.e. the package resource installs packages using apt on debian/ubuntuand using yum on centos/RHEL)!

Recipes

Recipes evaluate resources in the order they appear

package "pdns-recursor" do

action :install

template "#{node[:pdns][:confd]}/recursor.conf" do

source "recursor.cfg.erb"

owner "root"

group "root"

mode 0644

notifies :restart, "service[pdns-recursor]"

service "pdns-recursor" do

action [:enable, :start]

Recipes (2)

Recipes can include other resources, and are just ruby code

include_recipe "apache2"

%w{config logs files}.each do |dir|

directory "#{node[:myrecipe][:base_dir]}/#{dir}" do

recursive true

owner "myuser"

group "mygroup"

mode 02775

Cookbooks

Cookbooks are (shareable) packages for recipes.Cookbooks for chef ⇔ gems for ruby

cookbooks/python/

|-- attributes

| ‘-- default.rb

|-- files

| ‘-- default

|-- LICENSE

|-- metadata.rb

|-- providers

| |-- pip.rb

| ‘-- virtualenv.rb

|-- README.md

|-- recipes

| |-- default.rb

| |-- package.rb

| |-- pip.rb

| |-- source.rb

|-- resources

| |-- pip.rb

‘-- templates

‘-- default

‘-- profile_virtualenvwrapper.sh.erb

Cookbooks (2)

Currently 133 cookbooks available on opscode-cookbooks GitHub org.too many to list them all!

https://github.com/opscode-cookbooks

More cookbooks on the community site athttp://community.opscode.com/cookbooks

Metadata

maintainer "Opscode, Inc."

maintainer_email "[email protected]"

license "Apache 2.0"

description "Installs and configures mysql for client or server"

long_description IO.read(File.join(File.dirname(__FILE__), ’README.md’))

version "1.2.5"

recipe "mysql", "Includes the client recipe to configure a client"

recipe "mysql::client", "Installs packages required for mysql clients using run_action magic"

recipe "mysql::server", "Installs packages required for mysql servers w/o manual intervention"

recipe "mysql::server_ec2", "Performs EC2-specific mountpoint manipulation"

%w{ debian ubuntu centos suse fedora redhat scientific amazon }.each do |os|

supports os

Environments

Environments can be used to manage different environments (production,test, etc) in a single Chef setup.Roles can have different run list on different environments

name "production"

description "The production environment"

cookbook_versions(

"mysql" => "= 1.2.5", # use version 1.2.5 only

"apache2" => "~> 1.1" # anything 1.1.0 < x < 1.2.0

# default attributes for this environment

attributes(

"apache2" => {

"listen_ports" => ["80", "443"]

As with role, the ruby DSL gets compiled to JSON when uploading to server

Environments

name "production"

cookbook_versions(

attributes(

"apache2" => {

"listen_ports" => ["80", "443"]

Environments

name "production"

cookbook_versions(

attributes(

"apache2" => {

"listen_ports" => ["80", "443"]

Environments (2)

$ knife environment list

production

$ knife environment show production -F json

"name": "production",

"description": "The production environment",

"cookbook_versions": {

"mysql": "= 1.2.5",

"apache2": "~> 1.1"

"json_class": "Chef::Environment",

"chef_type": "environment",

"default_attributes": {

"apache2": {

"listen_ports": [

"override_attributes": {

Environments (3)

Cookbooks can be frozen, so that following uploads with the same versionwill fail.

$ # -E automatically sets a requirement for the specified environment

$ knife cookbook upload redis --freeze -E production

Uploading redis ...

upload complete

$ knife cookbook show redis 1.0.2 | grep "frozen"

frozen ?: true

$ knife cookbook upload redis

Uploading redis ...

ERROR: Version 1.0.2 of cookbook redis is frozen. Use --force to override.

ERROR: Failed to upload 1 cookbook.

$ knife environment show production | grep redis

users: = 1.0.2

Data Bags

Data bags provide an arbitrary store of globally available JSON data.

Data bags can be encrypted (but then cannot be searched, except for id)

Data Bags

Data bags provide an arbitrary store of globally available JSON data.

Data bags can be encrypted (but then cannot be searched, except for id)

Data Bags (Encrypted)

(warn: fake data ahead)

$ knife data bag show accounts gbagnoli

comment: tenoh >dieliSh ’i7eexeijeiSh^u9phaeGhuu4chaa *=

email: Ahr8is3ahChohm6aeneic(aef"ah1eereeVohhie6Up=

group: Waix8Pa#iniy#oh6eem$eij=

groups: xai7ong7aihiu1neH&ah3ier3Goh}rae7nik$einaeb=

id: gbagnoli

shadow: yie@jah0ve$g2AeGh}ido6koobuew|aebeenaequeRo(xaiYei8eizi+f

7ohqu <i@enequ&oh7ef -ahdae8dia[chah7ee4yie$N4EeBichee5eiro

h2JaGhae^k6aephohjahsh6Aeja^cheew}o)i0wo5iesish3dighiewoh

tohoh0eegho7eik=

shell: aeshi2ohy ,ai6ai\h2Ahquu=

ssh_keys: [..cut ..]

uid: thahvo2IGhoh3osho8Ees/a=

username: poh5WiuZ2Er:it!ee1ahf{u=

Data Bags (Decrypted)

$ knife data bag show accounts gbagnoli --secret -file ~/. chef/enc_db_secret

comment: Giacomo Bagnoli

email: [email protected]

group: wheel

groups: [asidev , users]

id: gbagnoli

organization: asidev

shadow: $6$ [... cut ...]

shell: /bin/bash

ssh_keys: ["ssh -rsa [... cut ...] Giacomo Bagnoli "]

uid: 3000

username: g.bagnoli

Anatomy of a Chef Run

• chef-client starts

• Builds node (runs ohai, perform deep-merge of attrs)

• chef-client registers with the server

• Cookbook sync

• Compiles resource collection, loading:• libraries• resources / definitions• attributes• recipes

• Executes - Configure Node• Converge: each resource is mapped to a provider and which takes

action on it• Saves Node• Runs notification Handlers

On errors, exception handlers are run.

• Cookbook sync

• Compiles resource collection, loading:

• libraries• resources / definitions• attributes• recipes

• Cookbook sync

• Compiles resource collection, loading:• libraries

• resources / definitions• attributes• recipes

• Cookbook sync

• Compiles resource collection, loading:• libraries• resources / definitions

• attributes• recipes

• Cookbook sync

• Compiles resource collection, loading:• libraries• resources / definitions• attributes

• recipes

• Cookbook sync

• Executes - Configure Node

• Converge: each resource is mapped to a provider and which takesaction on it

• Saves Node• Runs notification Handlers

• Cookbook sync

action on it

• Saves Node• Runs notification Handlers

• Cookbook sync

action on it• Saves Node

• Runs notification Handlers

• Cookbook sync

Chef development workflow

• Write cookbooks/recipe

• Upload the modified cookbook to the chef server

• Add the cookbook to a run list (in a node or in a role)

• Wait for chef-client to run on nodes

• Commit changes in git

Search

Full-text query engine based on Apache Solr.Searches can be performed from knife and in recipes.Almost any object is indexed by the chef server, like roles, nodes, apiclients and environments.

$ knife search node "recipes:apache2"

7 items found

$ knife search node "recipes:apache2 AND chef_environment:production"

5 items found

$ knife search node "roles:lxc_guest"

9 items found

$ knife search client "admin:true"

4 items found

$ knife search role "name:lxc*"

2 items found

Bootstrapping

Bootstrapping is installing chef on new nodes . . . using chef.First, create the node:

$ knife node create mynewnode.example.com

# .. fires up $EDITOR

# .. set run_list / attributes / etc / environment

Assuming that the new node is a bare ubuntu install, bootstrap the node

$ knife boostrap -N mynewnode.example.com -d ubuntu $NODE_IP --sudo -V -x ubuntu

Or, combine with provisioning (i.e. Amazon AWS)

knife ec2 server create -I ami -db595faf --flavor t1.micro --region eu -west -1 \

-G default -x ubuntu -N newnode.example.com -d ubuntu -Z eu -west -1a

Chef Omnibus bootstrap template

Bootstrapping

Monitoring: CheckMK and Chef

MonitoringWe use CheckMK to configure Icinga for monitoring our infrastructure.

CheckMK is a general purpose nagios plugin to retrive data from hosts.

1. One active check per host per check interval (calling check mk as aplugin).

2. The connection is done via TCP to the check mk agent on the targethost (All host data is sent back at once as ASCII text.)

3. check mk extracts performance data.

4. check mk checks warn/crit thresholds and submits results to Icinga aspassive checks.

2. The connection is done via TCP to the check mk agent on the targethost

(All host data is sent back at once as ASCII text.)

Monitoring - Nodes

On nodes, the check mk::agent recipe, included in all nodes via the base

role, installs the check mk agent.

It also sets up xinetd and the firewall so that connections to the agentare allowed only from the monitoring host(s).

Monitoring hosts are specified as attributes in the base role.

Monitoring - Nodes

Monitoring - Nodes (2)

(almost) Every cookbook pushes a MRPE or check mk plugin check tothe node, so the check mk agent returns data for all configured services.

i.e. , in the mysql :: server recipemrpe_check "mysql" do

script "check_mysql"

variables(

:passwd => node["mysql"]["server_root_password"],

:checks => checks,

:tunables => node["mysql"]["tunable"]

> telnet mysql-server.example.com 6556

<<<mrpe>>>

(check_mysql) mysql_idx 0 OK - index usage 53.60% | index_usage=53.60%;0:;0:

(check_mysql) mysql_running 0 OK - 0 long running processes | long_running_procs=0;10;20

(check_mysql) mysql_threads 0 OK - 18 client connection threads | threads_connected=18;80;95

Monitoring - Nodes (2)

(almost) Every cookbook pushes a MRPE or check mk plugin check tothe node, so the check mk agent returns data for all configured services.

i.e. , in the mysql :: server recipemrpe_check "mysql" do

script "check_mysql"

variables(

:passwd => node["mysql"]["server_root_password"],

:checks => checks,

:tunables => node["mysql"]["tunable"]

> telnet mysql-server.example.com 6556

<<<mrpe>>>

(check_mysql) mysql_idx 0 OK - index usage 53.60% | index_usage=53.60%;0:;0:

(check_mysql) mysql_running 0 OK - 0 long running processes | long_running_procs=0;10;20

(check_mysql) mysql_threads 0 OK - 18 client connection threads | threads_connected=18;80;95

Monitoring - Server

On the server, the check mk:: server recipe installs and configures icinga,nsca, nagvis, pnp4nagios smokeping and check mk.

The check mk configuration file (which is used by check mk to configureactive and passive checks in icinga) is managed as a template by therecipe. The recipe use the search API to discover nodes using aconfigurable query.

nodes = search(:node, node["check_mk"]["search_query"])

default query:

default [”check mk”][”search query”] = ”chef environment:production”

Monitoring - Server

default query:

Monitoring - Server

default query:

Monitoring - Server (2)

So, when a node is promoted to production, as soon as chef runs on themonitoring server,

• It adds the node to the check mk conf file (the template is the same,but data has changed)

• . . . the modified templates notifies the check mk write conf

resource, which is queued

template "#{node[’check_mk’][’conf_dir’]}/main.mk" do

source "check_mk_main.mk.erb"

variables(

:nodes => nodes,

:hostgroups => hostgroups,

:params => node[’check_mk’][’params’]

mode 0644

owner "root"

group "root"

notifies :run, "execute[check_mk_write_conf]"

variables(

:nodes => nodes,

mode 0644

owner "root"

group "root"

variables(

:nodes => nodes,

mode 0644

owner "root"

group "root"

variables(

:nodes => nodes,

mode 0644

owner "root"

group "root"

The check mk write conf resource regenerates icinga conf

execute "check_mk_write_conf" do

command "#{node[’check_mk’][’prefix’]}/bin/check_mk -O"

action :nothing

Then the recipe scan nodes for services (using check mk inventory)This is done only the first time (no reinventory)

nodes.each do |n|

if not n[’tags’] or not n[’tags’].include? "noagent"

check_mk_inventory n[’fqdn’]

check mk inventory is an LWRP defined in the check mk cookbook

action :nothing

nodes.each do |n|

action :nothing

nodes.each do |n|

That way new nodes are automatically added to the monitoring when theyshow up as result of the search query.

Since cookbooks push and configure check mk plugins, everything getsmonitored andall the monitoring logic is in the cookbook itself.

Monitoring Chef Clients

Once you start relying on chef, you want to know if chef-client iscorrectly running on nodes and if/when it fails.

Instead on relying on the active model of check mk, we use a completelypassive approach. We use NSCA for this.

When chef-client runs on the node, it pushes a report/exceptionhandler called NSCAHandler.This handler runs at the end of a chef-client run, and submits theresult to the NSCA server on the monitoring host, so that:

• check is in critical state if chef run failed.

• check is in warning/critical state if time elapsed is above thresholds

• check is in warning/critical state if the number of modified resourcesis above thresholds

Monitoring Chef Clients (2)

Passive checks are created on the server for every node which runs thechef-client

These checks have a freshness threshold of 1 day, so that if no dataarrives from the node the check will enter the UNKOWN state, meaning thechef-client is not running on that node.

Monitoring - Final Words

The check mk:: server recipe also configures:

• host parents

• host groups

• service groups

• notification periods for services

• contact groups and administrators (data is in the accounts data bag)

• uses smokeping to perfom hosts checks (instead of using check ping)

• . . . and integrates smokeping web ui with check mk multisite.

• host icon for the status map :-)

Behavior can be changed by setting nodes tag with knife. i.e, this changesthe notification period for a host.knife tag create myhost.example.com workhours

Questions?

Bagnoli Giacomo

[email protected]/@gbagnoligithub.com/gbagnolibitbucket.org/gbagnoligplus.to/gbagnoli

Thank you!

Intro to Chef

Technology

Transcript of Intro to Chef