Introduction to Systems Management

with SaltStack

Craig Sebenik Infrastructure Engineer@Matterport

3 Oct 2015

• Introduction

• Architecture Overview

• Execution Modules

• States

• Data - Minion and Master

• Extending Salt

• Demo

• Summary

What Is SaltStack?• System and Configuration Management

• Encrypted communication channel

• Remote execution framework

• 100% open: one of the most active on github

• Scales to tens of thousands of nodes

• Built (and extended) with python

What Am I Covering?• Simple/quick overview of salt

• Very simple examples

• Only a basic single master topology

• Only the core functionality

• Glossing over details: ask questions!

Who Am I?• Degrees in Chemistry, Mathematics, Food

• Computational Chemist (lifetime ago)

• Abbott Labs, Eastman Kodak, Parke-Davis

• Sysadmin/SRE

• NetApp, LinkedIn, Matterport

One More Thing About Me…

Quick History Of Salt• Initial release in March 2011

• States added a few months later

• Pillars added March 2012

• Salt SSH added Sept 2013

• Salt Cloud merged in Jan 2014

• Custom transport (RAET) added in Jul 2014

• Introduction

• Architecture Overview

• Execution Modules

• States

• Data - Minion and Master

• Extending Salt

• Demo

• Summary

Minions and Master• Master: central command and control

• Minion: paired with master

• Encrypted communication

• Communication over ZeroMQ using MessagePack

• Target minions based on their attributes

Quick Example

What Does This Mean?

• Central management of many nodes

• Ensuring hosts match a “recipe”

• Easy to add more hosts that match a template

PUB-SUB

master minion1

minion2

4505

Returning Data

master minion1

minion2

4506

Targeting Minions• List: “minion1,minion2

• Globs: “minion*”

• Regular expression: “minion([2|3])”

• Grains: “OS: Ubuntu”

• Combinations of the above

Set up Trust With salt-key• Salt uses standard public key encryption

• Key exchange

• Master needs to verify identity of minions

• User needs to “accept” the minion’s key

• Minion’s public key stored on master

• Master’s public key stored on minion

• Introduction

• Architecture Overview

• Execution Modules

• States

• Data - Minion and Master

• Extending Salt

• Demo

• Summary

Execution Modules• Salt comes with over 100 modules

• Over 1000 functions

• Examples:

• pkg.install, pkg.remove

• file.copy, file.find, file.chown

• user.add, user.info

Minor Vocabulary Clarification

• Modules contains functions

• Modules correspond to python files

• Functions correspond to methods

• There are some exceptions, but beyond today’s scope

Add User To All Hosts

What’s Happening• Master looks at target (‘\*’) and determines hosts

• Puts message out on event bus

• Over ZeroMQ using messagepack

• Minion sees message and executes

• All execution is on minion, not master

• Minion returns data back to master

Master Maintains Job Data

• Job cache on master

• Contains history of jobs run and data returned

• Tools to query the job cache

• Default is to cache 24 hours of history

• Performance penalties when storing longer

Commands Sent In Parallel

• Command sent via event bus

• Minions see and execute

• Jobs are done asynchronously

Can Run Locally

• Command to run locally: salt-call

• No central coordination

• Data *IS* still returned to master

• Can bypass with “—local” flag

Documentation

• Function called “sys.doc”

• Uses python docstrings

• Important when writing your own custom modules/functions

salt-call Example

• Introduction

• Architecture Overview

• Execution Modules

• States

• Data - Minion and Master

• Extending Salt

• Demo

• Summary

States• Recipe for how a host should be configured

• Default file format is YAML (with jinja)

• Write state files on the master

• Master will sync to minion automatically

• States use the remote execution framework

• But, they are not the same

State Example

Running State Example

Running highstate

• Running individual states can be tedious

• Collect all states for a host (or “template”) in a single file: top.ls

• Called: top file

• Target just like running the “salt” command

Example Top File

Running highstate

Targeting Example

Running Targeting Example

• Introduction

• Architecture Overview

• Execution Modules

• States

• Data - Minion and Master

• Extending Salt

• Demo

• Summary

Data: Minion and Master

• Grains: minion side data

• Example: host operating system

• Pillars: master side data

• Example: database passwords

Grains: Minion-Side Data• Data gathered on the minion

• Master has a cache of minion grains

• Salt comes with a number of grains built in

• OS name (eg CentOS)

• number of CPUs

• kernel version

Viewing Grains

Targeting With Grains

Adding Grains• Minion config

• /etc/salt/grains

• Via command

• sudo salt minion grains.setval foo bar

• Via python (will discuss later)

Pillars: Master-Side Data

• Data sent to a specific minion (from master)

• Typically used for sensitive data

• E.g. passwords

• Uses a “top file” (just like “states”)

Pillar Example

Running Pillar Example

Targeted Pillar Data

Running Targeted Pillars

• Introduction

• Architecture Overview

• Execution Modules

• States

• Data - Minion and Master

• Extending Salt

• Demo

• Summary

Extending Salt• Jinja

• Custom modules/functions (python)

• salt python API (LocalClient)

• Customizations are synced via salt command

• Easy to automate

Templates Using jinja

• Jinja is a widely used python templating language

• Inspired by Django’s templates

• Default template for flask applications

• Gives basic control commands to flat files

Jinja Example

Running The ‘vim state’

Python Module/Function

Custom Modules are NOT Automatically Synced

Running hello.world

Docstrings Are Important

• Introduction

• Architecture Overview

• Execution Modules

• States

• Data - Minion and Master

• Extending Salt

• Demo

• Summary

Demo Minions

• minion1: development database server

• minion2: development application server

• minion3: production database server

• minion4: production application server

• Introduction

• Architecture Overview

• Execution Modules

• States

• Data - Minion and Master

• Extending Salt

• Demo

• Summary

Summary• Master and minions encrypted communications

• Grains: minion-side data, Pillars: master-side data

• Execution functions run on the minions

• States are formulas/recipes to define a host

• Collect multiple states with highstate

• Lots of ways to extend salt functionality

Other Features• Runners: master side orchestration

• Orchestrate Runner: master coordination of states

• Salt cloud: manage cloud virtual machines

• Salt ssh: like normal salt without minion process

• More advanced topologies

• multi-master

• master-less minions (with salt-call)

• GitFS

References

• https://docs.saltstack.com/en/latest/

• https://docs.saltstack.com/en/getstarted/

• https://github.com/saltstack/salt

Questions?

@craigs55 irc:chitown

https://www.linkedin.com/in/craigsebenik

Yes, we’re hiring! http://matterport.com/positions/