RansackAn Application Built on Ansible’s API

Paul Durivage• DevOps Engineer at Rackspace

• Ansible Contributor

• Aspiring Pythonista

• Army Veteran

@angstwad

angstwad

Rackspace

• Dedicated Hosting

• Cloud Hosting

• *aaS Offerings

• Fanatical Support

DevOps from Scratch

• Shell Scripts

• Fabric

• Cook*

*This is a pseudo name for prominent config management tool

Our team of three was formed to figure out exactly how to be “DevOpsy”. The goal was to write applications that we thought added value to the business unit and for our customers, to consume all the latest and greatest products and to give our experience back to customers. Is this everyone’s story in the room? Eventually, we had many web applications and needed to figure out how to get them all deployed. We started with simple shell scripts and some Upstart jobs — it sucked. We used Fabric because it was familiar, and filled in all the gaps weren’t well-solved by mere shell. Along came “Cook”…

Cook* made me feel stupid.

*This is a pseudo name for prominent config management tool

I took the two-day course and wasn’t able to accomplish anything by the time I was done. Cook made me feel stupid. I knew this was not the way forward.

Ansible

THE SOLUTION TO ALL OUR PROBLEMS. !We could use this everywhere — automate everything.

The Problem (is not Ansible)

Tickets come in. Customers ask us to manage their servers: add this user, add this package, configure this vhost, fetch this log file. Some of this is tedious, boring work. Rackspace has many Sys Ads — smart people. Accelerating their work pleases customers and saves money. How do we automate away tedious, repeatable, error-prone work?

Example: I want to add a user to a number of machines…

Jim writes a shell script…

SysAd Jim writes a shell script that adds a specific GID, adds a user, then updates the password. Using current tools, this script runs serially and takes about an hour to work on 100 machines.

Bob writes a shell script….

Bob writes a script that does something similar, but adds some SSH key functionality into his.

Billy wrote one, too…

Add a user with a password with some sudo privs.

Oh, great, another script…

This user attempts to change the password of an existing user, and then does some weird grep stuff to find a username inside files on the root partition.

Now we have 4 shell scripts which attempt (more or less) to do the same thing. All were written independently of one another, vary wildly in functionality, offer pretty poor validation, none incorporated testing… !We need to write fewer shell scripts. So how do we do that?

Let’s Multiplex

LET’S NOT

The time it takes to set up all these sessions is wasted; every keystroke is essentially gone, unless we want to scrape shell history files, record sessions, or use some other magical, nonsensical tool. The work isn’t repeatable! This is not the answer!

We can do better.

We have programming languages and APIs, secure connection protocols, and software. Why are we working so hard to manage systems?

We can do better.

Ransack

So we built Ransack.

Ransack

• Uses internal Rackspace APIs

• Implements Ansible API

• Creates a new CLI

Ransack is custom Ansible. It’s the paring of custom inventory scripts, to gather information about our customers, with a CLI that is more appropriate for our workflow.

Ansible• Extensible

• Inventory

• API

• Modules

• Parallelization

• Consistency

• Idempotency

• Community

So let’s talk about why Ansible makes sense for this by talking about some of the features it provides. !For time, we’re skipping over a lot of the reasons why we chose Ansible. :) !

Inventory• What Is It?

• Why Start With Inventory?

• Two types

• Static

• Dynamic

What is inventory? It’s “stuff” to be managed. Servers, VMs, containers. We started with inventory because it’s the first logical step to getting Ansible to do work on Rackspace machines. There are two types, static and dynamic, but we’re primarily concerned with getting data at runtime which is all about dynamic inventory.

Dynamic Inventory!

• Talk to APIs

• Create Ansible-specific data

• Serialize data

• Caching

• Logically group hosts

Scripts, create data, talk to APIs, serialize to JSON, while logically grouping hosts

Ransack Inventory

$ ACCOUNT=12345 ransack-inventory-core --list$ ACCOUNT=cloud_acct ransack-inventory-cloud --list

Running the scripts manually.

Ransack Inventory

"community": [ "123456-community.res.rackspace.com"],"corpvm": [ "123456-corpvm1.rackspace.com"],"db": [ "db1.prod.rackspace.com", "db2.prod.rackspace.com"]

Ransack Inventory

!$ ACCOUNT=8271 ansible-playbook -i \ ransack-inventory-core site.yml

Ansible commands and playbooks work!

Having inventory scripts available means that anyone comfortable with Ansible can go ahead and run any playbook using Rackpsace inventory.

Ansible• Extensible

• Inventory

• API

• Modules

• Parallelization

• Consistency

• Idempotency

• Community

That’s inventory. Now it’s time to talk about the Ansible API. !

API

Ansible’s first commit. API usage is equal citizen to CLI. REMOTE EXECUTION LIBRARY.

API• Classes

• ansible.inventory.Inventory

• ansible.runner.Runner

• ansible.playbook.PlayBook

• References for use

We are deep in code. Ransack uses the Inventory, Runner, and Playbook classes. Basically we hook in at the core of the API, gather inventory from our dynamic inventory scripts, and then instantiate a Runner or Playbook to go out and do the work. The “ansible” command is the reference implementation of the API, though there are some docs to get you started. !!

RansackSane Defaults for Ansible

The CLI provides SANE DEFAULTS for Ansible in order to play nicely with our customer’s SSH configurations, our internal networks. Suppress warnings (like ControlPersist messages) Implement best practices (always SSH, no pipelining, longer timeout) Play nicely with our changes (modules, scripts)

Ransack CLI

!$ ransack-core --args ACCOUNT LIMIT MODULE --module-args$ ransack-cloud --args ACCOUNT LIMIT MODULE --module-args

The API is used so we can provide a “custom” interface that provides reasonable arguments like account information, inventory/limit specs, modules and module args.

Ransack CLISelf-Documenting

And we wanted our API to be self-documenting. Problematic because we had to carefully pick out a subset of modules to wrap argparse around, so that we could document like this. In the future we’ll probably parse module documentation dynamically to make every module (with limited exceptions) available.

Ransack CLI

!ransack-core --as-root 8271 334141 user --name bob \ --password --update-password always

Command example adding a user to account 8271 on device 334141. Notice the —as-root flag to keep SysAds from worrying about HOW to escalate privileges.

Ransack CLI

Done.

Launch Day

We launched and immediately had problems — almost all of them related to the installation process. !Installation was drastically impacting usage.

Installation

We began installation first by offering various shell scripts that would set up a virtualenv, install Ransack. Later this was consolidated into the “super mega installer” to cover OS X, Gentoo, Ubuntu, EL 6, Fedora, Debian, and Arch. This method sucks, primarily because of pyrax dependencies (PBR) dependencies, issues with versions of pip and distribute.

Installation

Virtual machines: we created two images, one a TinyCore Linux VM which is 92 MB, and Ubuntu 12.04, which is 620 MB. We control the environment and ship an appliance. Works really well.

Installation

Next we created a Docker image that could be launched on a local machine without the need to install a heavy VM or Hypervisor. This is problematic, however, because Docker usage across our SysAd audience is still pretty low.

The Future of Ransack

Ransack as a Service Dynamically pick up all modules Deploying more often (We use Drone for CI/CD today) API/middleware

I wanted to talk about…• Packer

• Docker

• SSH

• ssh-agent

• SSH “Bastions”

• Drone

• Testing

• Issue Tracking

• Data collection

• Updates

but I ran out of time.

QuestionsAnd Obligatory Applause