TESTING SERVERS LIKE SOFTWARE

MEDEVELOPER TURNED ops guy

WORKING AT KAINOS, CONTRACTING ON GOVERNMENT PROJECTS

PREVIOUSLY ON THE IER PROJECT

NOW LIVE! HTTPS://WWW.GOV.UK/REGISTER-TO-VOTE

NOW ON THE DEFRA CAPD PROJECT

FOREWARNING▸ Ruby and Puppet Biased

▸ They're the tools I use the most!▸ But most tools mentioned are system and tool agnostic

SO YOU'VE MADE A CONFIGURATION CODE

CHANGE...

THE UNIT TESTS PASS...

IT'S BEEN CODE REVIEWED

SO YOU PUSH TO PRODUCTION!

IT DOESNT WORK...

WHAT'S THE MISSING STEP?

ACCEPTANCE TESTING:Serverspec

"Serverspec tests your servers' actual state through SSH access"

CHECK YOUR SERVER▸ Is $package installed?▸ Does file contain $foo?

▸ Does the firewall have the correct rules?▸ Is service running?

▸ etc...

RESOURCE TYPEScgroup, command, cron, default_gateway, file,

group, host, iis_app_pool, iis_website, interface, ipfilter, ipnat, iptables,

kernel_module, linux_kernel_parameter, lxc, mail_alias, package, php_config, port, ppa,

process, routing_table, selinux, service, user, windows_feature, windows_registry_key, yumrepo,

zfsfs

MOST ARE FAIRLY SELF

EXPLANATORY

return_stdout

describe command('cat /etc/resolv.conf') do it { should return_stdout /8\.8\.8\.8/ }end

content

describe file('/etc/httpd/conf/httpd.conf') do its(:content) { should match /ServerName www.example.jp/ }end

MY BREAD AND BUTTER

BE_RESOLVABLEdescribe host('serverspec.org') do it { should be_resolvable }end

describe host('serverspec.org') do it { should be_resolvable.by('hosts') }end

describe host('serverspec.org') do it { should be_resolvable.by('dns') }end

BE_REACHABLEdescribe host('target.example.jp') do # ping it { should be_reachable } # tcp port 22 it { should be_reachable.with( :port => 22 ) } # set protocol explicitly it { should be_reachable.with( :port => 22, :proto => 'tcp' ) } # udp port 53 it { should be_reachable.with( :port => 53, :proto => 'udp' ) } # timeout setting (default is 5 seconds) it { should be_reachable.with( :port => 22, :proto => 'tcp', :timeout => 1 ) }end

FULL SPECS FOR A WEB SERVERrequire 'spec_helper'

describe package('apache2') do it { should be_installed }end

describe service('apache2') do it { should be_enabled } it { should be_running }end

describe port(80) do it { should be_listening }end

EXAMPLE:HTTPS://GITHUB.COM/JVOORHIS/

VAGRANT-SERVERSPEC

LIVE DEMO TIME: SERVERSPEC!

OUR WORKFLOW:CHANGE IN PUPPET MADE =>TESTED IN VAGRANT INSTANCE

CODE REVIEWED, PASSES CI AND MERGEDPUSHED TO INTEGRATION ENVIRONMENT

SERVERSPEC TESTS RUN ON INTEGRATION ENVIRONMENTANY ISSUES: FIX OR IF BIG ENOUGH, REVERT

CONTINUES DOWN THE PIPELINE TO PRODUCTION

SERVERSPEC:PRETTY NEAT

BUT THERE'S ALSO LANGUAGE SPECIFIC TOOLS!

TEST KITCHEN

▸ Uses serverspec as it's core▸ It's basically helper wrappers to install chef

▸ And run the cookbooks given▸ Used for cookbook acceptance testing

EXAMPLE:HTTPS://GITHUB.COM/OPSCODE-

COOKBOOKS/APT/

LIVE DEMO TIME: TEST-KITCHEN!

BEAKER

▸ Again: serverspec as it's core▸ But specs customised with Puppet references

▸ Specific rspec grammer around running manifests etc.

EXAMPLE:HTTPS://GITHUB.COM/PETEMS/PUPPET-

SWAP_FILE

require 'spec_helper_acceptance'

describe 'swap_file class', :unless => UNSUPPORTED_PLATFORMS.include?(fact('osfamily')) do

context 'swap_file' do context 'ensure => present' do it 'should work with no errors' do pp = <<-EOS class { 'swap_file': } EOS

# Run it twice and test for idempotency expect(apply_manifest(pp).exit_code).to_not eq(1) expect(apply_manifest(pp).exit_code).to eq(0) end

context 'custom parameters' do it 'should work with no errors' do pp = <<-EOS class { 'swap_file': swapfile => '/tmp/swapfile', swapfilesize => '5 MB', } EOS

it 'should contain the given swapfile' do shell('/sbin/swapon -s | grep /tmp/swapfile', :acceptable_exit_codes => [0]) shell('/sbin/swapon -s | grep 5116', :acceptable_exit_codes => [0]) end end endend

I KNOW WHAT YOU MIGHT BE THINKING...

Wait, whats the difference between this and monitoring?

MONITORINGKEEP THE LIGHTS ON▸ "Fix me now!" issues

▸ Dynamic changes - Things crashing, hard drives are full

ACCEPTANCE"MY CHANGES DIDN'T BREAK ANYTHING"

▸ Smoke tests▸ Human readable

▸ One-off for edge-cases▸ Auditing

HOWEVER

USING SERVERSPEC AS MONITORING IS POSSIBLE!

AND HAS SOME PRETTY SWEET BENEFITSHTTP://WWW.SLIDESHARE.NET/M_RICHARDSON/SERVERSPEC-AND-SENSU-TESTING-AND-MONITORING-

COLLIDE

Q&A

LINKShttp://serverspec.org/

http://vincent.bernat.im/en/blog/2014-serverspec-test-infrastructure.html

https://github.com/serverspec/serverspec

http://www.debian-administration.org/article/703/A_brief_introduction_to_server-testing_with_serverspec

"ChefConf 2014: Gosuke Miyashita, "Serverspec: The Simplest Server Testing Tool Ever" - http://www.youtube.com/watch?

v=6GvlHImeloo*