Testing Salt States: Part 1The Basics

About Me (Jason Denning)

• DevOps Engineer• SaltStack user for ~ 3 years• SaltStack Certified Engineer #2• AWS Certified Architect• OSS Advocate (first Linux install ~ 1998)• Pythonista• jasondenning on GitHub / @jason_denning on Twitter

● Provide analytics and marketing/segmentation tools for mobile app developers

● Hosted backend APIs available world-wide

● Lots of traffic● Big fans of SaltStack● upsight.com

WARNING: Work In Progress

● Testing infrastructure is not a common practice ● These slides are rough

● Your Mileage May Vary!

● We should work (together) on improving this

Why Test?

● DevOps: Infrastructure is code

● Code should be tested

● ⇒ Infrastructure should be tested

Why Test? (continued)

● Find errors faster

● Prevent regressions

● Allow others to contribute ● Simplify Code Reviews (CI)

● Improve Documentation

What to Test

● Basic Syntax (linting)

● Individual States

● State Relationships ● Different Environments

● As much as possible (within reason)

Testing Basics

● Tests should run quickly (and automatically)

● Tests should be isolated - know what you’re testing! ● Unit Tests: Test a single ‘unit’ of code

● Integration Tests: Test interactions between units

● Acceptance Tests: Test that code does what we want

Minimal Testing

$ salt-call state.highstate test=True

● Better than blindly applying states

● Still not what we’re looking for○ Manual○ Slow○ No isolation○ Error-prone (“Oops, I

didn’t see the red line!”)

First Improvement

Test states individually:

$ salt-call state.sls foo test=True

● Slight improvement○ More isolation○ Faster than highstate

● Still not what we want

Testing Basic Syntax of SLS Files

● First Attempt: Verify SLS files are valid YAML○ Problem: Only works for very basic states - Jinja breaks the test!○ Problem: What about other renderers?

● ● Improvement: Test that Salt can render the SLS file

○ “salt-call state.sls foo test=True” works for this○ Alternative: “salt-call state.show_sls foo”

Testing Basic Syntax

$ salt-call state.sls foo test=True(with syntax error)

$ salt-call state.show_sls foo(with syntax error)

Note: Better error message!

Testing Basic Syntax

$ salt-call state.show_sls foo(no syntax error)

Testing Basic Syntax - Notes

● Salt built-ins are useful!

● Better: A separate tool for validating syntax (somebody should build one!)

● Don’t forget about outputters:

● salt-call state.show_sls foo --output=json

Testing in Isolation

● We need to be sure that test results are not influenced by previous tests

● Unit testing frameworks handle this automatically

● Ideal: Create a new, clean infrastructure for each test○ Problem: Tests must run quickly!

● Solutions:

○ Use VMs (still to slow if we’re running lots of tests)○ Use Containers (e.g. LXC / Docker)

Docker Containers vs. Full VMsDocker Pros:

● Easy● Quick to build and destroy

Docker Cons:● Missing Functionality (testing services will take

more work)● Linux only

VM Pros:● Full OS install - services work / any OS● Easy to manage if using tools such as Vagrant● Might be identical to your actual environment

VM Cons:● Slow(er) to build and destroy

Best Practice: Use both! (For different types of test)

Docker: Quickstart● Docker makes it easy to build & manage LXC containers● Allows one to quickly customize existing container images● Mount host directories in the container

Basic Dockerfile (~/saltpdx/Dockerfile):

FROM phusion/baseimageRUN sudo add-apt-repository -y ppa:saltstack/saltRUN sudo apt-get updateRUN sudo apt-get -y install salt-minionVOLUME ["/etc/salt", "/salt/states", "/salt/pillar"]CMD salt-call state.highstate

● Build an image (tagged salt/test, version 1):$ docker build -t ‘salt/test:1’ .

● Start a container using the image:$ docker run --rm -it -v ~/saltpdx/salt:/etc/salt -v ~/saltpdx/states:/salt/states -v /

~/saltpdx/pillar:/salt/pillar salt/test:1

Docker Quickstart● Docker creates a container

based on the image

● Host directories are specified using the -v flag

● Docker starts the container and runs the CMD specified in the Dockerfile

● --rm flag tells docker to delete the container after the command has run

● -it (or -i -t) for interactive terminal mode (add ‘/bin/bash’ to end of command to get a shell)

Docker Tips

● Mount host directories as volumes for:○ Salt configuration○ States○ Pillar○ Logs?

● Configure containers as salt-masterless to avoid networking & PKI issues

● Use the --rm flag to automatically delete container instances after they’ve run

● Tag & version images

Docker: Entrypoint● When you run a Docker container, Docker executes a binary called ENTRYPOINT● The CMD that you pass to Docker (via Dockerfile or the cli) is passed as an argument to

the ENTRYPOINT● By default, Docker will use “/bin/sh -c” as the ENTRYPOINT - this can be overridden● e.g.:

$ docker run --rm --entrypoint /bin/echo salt/test:1 foo

● We can exploit this to make a simple test runner● Dockerfile:

FROM phusion/baseimageRUN sudo add-apt-repository -y ppa:saltstack/saltRUN sudo apt-get updateRUN sudo apt-get -y install salt-minionVOLUME ["/etc/salt", "/salt/states", "/salt/pillar"]ADD test_salt.sh /test_salt.shRUN chmod +x /test_salt.shENTRYPOINT ["/bin/bash", "/test_salt.sh"]

Docker: Entrypoint (con’t)

● test_salt.sh:

Docker: Entrypoint (con’t)After building a new image, we can run:$ docker run --rm -v / ~/saltpdx/salt:/etc/salt -v / ~/saltpdx/states:/salt/states -v / ~/saltpdx/pillar:/salt/pillar salt/test:2 / highstate

Which will start a container, run “salt-call state.highstate”, and exit.

Docker: Entrypoint (con’t)Or, we can run:$ docker run --rm -v / ~/saltpdx/salt:/etc/salt -v / ~/saltpdx/states:/salt/states -v / ~/saltpdx/pillar:/salt/pillar salt/test:2 / foo

Which will run “salt-call state.show_sls foo”, then “salt-call state.sls foo”, and exit.

Next Step: Automation

● We’ve got the basic tools - the next step is to add some automation● Host-side automation:

○ Starting and destroying containers○ Determine what states to test○ Collect & summarize test results○ Test various permutations (different Linux version, different

grains, etc.)

● Container-side automation○ Discover & test sub-components (e.g. file templates)○ Integration/Acceptance/Behavioral Tests

Automation Overview

● Testing is initiated by running a test-controller script on the host● The test-controller should:

○ Generate a list of states to test and environment permutations○ Start a container and trigger the test runner○ Collect test results○ Destroy the container○ Repeat for each state

● The Docker image should have a test-runner script that is used as the ENTRYPOINT

● The test-runner script should:○ Determine specific tests to run○ Run the tests○ Output Results

Next Time - Part 2

● Actual code for these automation scripts● Behavioral Testing (for acceptance/integration tests)● Collecting and Summarizing Results in a useful fashion● Continuous Integration● Adding Vagrant VMs for additional tests● Testing multi-machine infrastructures● …???

Thanks!

Discussion / Questions?

salt@jasondenning.comGitHub: jasondenningTwitter: @jason_denning