Scalability and Consistency

The Nitty Gritty

Overview

• The only sane way to scale is to ensure consistency and automation

• Build things in a way in which you don't care about individual servers (don't be afraid to rekick a server)

• You should never need to login to an individual box once it's in production 

How to Accomplish This

• Use an automated provisioning tool (such as kickstart/cobbler)

•  Use configuration management (such as puppet)•  Test, test, test

Users and Authentication

• Logging in as root directly is bad• Keys used for automated processes should be

separate from user keys• Access to a service on a box and SSH access to the

box should not be linked

Packaging

• Use a local copy of distribution repositories• Do not use external repositories directly• Use RPMs (do not use PECL, PEAR, CPAN, gem,

extracting tarballs, compiling from source, etc.)

External repositories are moving targets. Depending on when a server is provisioned, it could end up with different package versions.

Non-RPM packaging methods don't have upgrade, uninstall, and versioning, which is why they should be avoided.

Building RPMs

• It's perfectly fine to build a custom RPM when one is not readily available

• Don't rebuild just to change file locations (such as putting stuff under /box)

• Build in a "sterile" environment• Make sure RPM spec files list all of their direct build

and runtime dependencies

General Puppet Tips

• Write modules/classes in a way that they can be reused

• Be specific...don't assume something will happen a certain way if you don't define it

• Follow the Puppet Style Guide• Don't hard-code anything (hostnames in particular)• Use dependencies

http://docs.puppetlabs.com/guides/style_guide.html

Puppet Tips - Users

Use this:

group { "bob":  ensure => present,  gid      => 503,}

user { "bob":  ensure => present,  gid       => 'bob',  require => Group["bob"],}

Instead of:

user { "bob":  ensure => present,}

• Manage the group and the user• Specify the UID and GID to ensure consistency

across servers

Puppet Tips - Packages

Use this:

package { "php-pecl-memcache":  ensure => installed,}

Instead of:

exec { "install_pecl_memcache":  command => 'pecl install memcache',}

• Use a single package manager (RPM) as much as possible

• Specify versions when it makes sense

Puppet Tips - Dependencies

Use this:

package { "nslcd":  ensure => installed,}

service { "nslcd":  ensure => running,  require => Package["nslcd"],  subscribe => Package["nslcd"],}

Instead of:

package { "nslcd":  ensure => installed,}

service { "nslcd":  ensure => running,}

• Dependencies ensure that puppet performs operations in a particular order

Puppet Tips - Testing

• Test your changes on a non-production server• For larger changes in puppet, it's not completely

tested until you can build a server from the ground up and puppet sets everything up properly without manual intervention

• If you get errors on the first puppet run and none on the second, add more dependencies

Puppet Tips - Roles

• Instead of defining everything on a per-node basis, define roles and put servers into them. It's better to define 500 roles and have servers fit neatly into them than to have 1000 node definitions, each of them differing by only a little bit

• All roles should be defined in the "nodetype" module and inherit from nodetype::generic

• This will allow us to eventually use an ENC (external node classifier)