Pilot Tech Talk #10 — Practical automation by Kamil Cholewiński
31
Practical automation
Transcript of Pilot Tech Talk #10 — Practical automation by Kamil Cholewiński
Practical automation
Automation?
For whom?For DevOps and IT...
...and for anyone who can code
...and for mere mortals as well!
How?With Ansible, Docker, Hubot...
...and with whatever else you see fit
Why automate?
Humans love being lazy
Do the extra work now
Be lazy forever after
Most things are already automated
Except for those you really need
Engineer: a person that takes 3 hours to turn a 2-hour task into a 1-hour task
Reap benefits until heat death of the universe
(Or until things break mysteriously...)
"If it's worth doing it twice, it's worth automating it"
Except when it's not
Often, attempting automating things can get you nowhere and waste time
So when is it a good idea to automate?
Anything repetitive
Create / delete user accounts, rotate passwords...
Upgrade vulnerable software packages...
Tweak a config file...
...on ALL the servers!
...all. the. time...
Anything daunting
Go through all email groups and delete those that have no members
Migrate data from an old service to a new one
Any low-hanging fruit
Things that are not hard, not too boring; still easy to script
Good ROI, easy to get into the habit of automating
What if something can't be automated?
Delegate it!
Maybe there's someone more competent?
Maybe it should be a person's decision, not a machine's?
Maybe you can replace yourself with a push-button, and give the button to the guy incharge?
Think of a better process - talk to your fellow humans
Automation: For devops(Where devops = developers and sysops working together.)
Problem: Managing server configuration
My story:
Developer suddenly turned sysadmin; inherited entire company's web hosting &server infrastructure; ca 30-40 servers
Nothing was documented, everything was done by hand
Started an inventory in a spreadsheet; quickly realized it doesn't scale, and mostimportantly: it's not machine-readable
First attempt: Ansible
Server / configuration management tool
Installed Ansible, re-created the inventory as .ini file, now I can ping ALLhosts with one command:
ansible -m ping all
Yes, this is the way to go!
git commit -m "Hello, world"
We kept the repository ("The Cookbook") private
Lots of valuable, reusable code...
...But many parts tightly integrated with our infrastructure
2 years, 1800+ commits, 300+ pull requests, and 8 contributors later...
How do we use Ansible? What can it do foryou?
Need a new server or two...
ansible-playbook cloud-ec2.yml -e project=kittens -e env=live -e count=2
Provision the base system? Simple:
ansible-playbook provision-base-system.yml --limit tag_project_kittens
Deploy project "kittens" to staging environment:
ansible-playbook deploy-kittens.yml -e env=stage
Update our DNS zones:
ansible-playbook managed-dns.yml
Audit all servers for access; remove revoked public keys, etc:
ansible-playbook managed-cleanup.yml
How does a typical playbook look like?--- - hosts: managed vars: docker_opts: "" handlers: - name: Restart Docker daemon service: name=docker state=restarted - name: Refresh apt cache apt: update_cache=yes - name: Reload systemd daemon shell: /bin/systemctl daemon-reload tasks: - name: Add docker apt key apt_key: keyserver: pgp.mit.edu id: "58118E89F3A912897C070ADBF76221572C52609D" tags: install - name: Install apt-transport-https apt: name=apt-transport-https tags: install - name: Add docker apt sources copy: content: | deb https://apt.dockerproject.org/repo {{ ansible_distribution.lower() }}-{{ ansible_distribution_release }} main dest: /etc/apt/sources.list.d/docker.list notify: Refresh apt cache tags: install
- name: Flush handlers meta: flush_handlers - name: Create /etc/apt/preferences file: dest: /etc/apt/preferences state: directory tags: install - name: Pin Docker version copy: dest: /etc/apt/preferences/docker content: | Package: docker-engine Pin: version 1.12.* Pin-Priority: 1100 tags: install - name: Install Docker apt: name=docker-engine notify: Restart Docker daemon tags: install - name: Install Docker Python goodies pip: name={{ item.name }} version={{ item.version }} executable=pip2.7 environment: PATH: /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin with_items: - { name: "docker-py", version: "1.9.0" } - { name: "docker-compose", version: "1.8.0" } tags: install - name: Create docker group group: name: docker state: present notify: Restart Docker daemon tags: install - name: (systemd) Tweak docker daemon flags when: docker_opts copy: dest: /etc/systemd/system/docker.service content: |
content: | [Service] ExecStart= ExecStart=/usr/bin/docker daemon -H fd:// {{ docker_opts }} \n notify: - Restart Docker daemon - Reload systemd daemon tags: install - name: Flush handlers meta: flush_handlers
Problem: Is Ansible good or bad?
Better than doing it by hand? Definitely!
Better than curl | sudo bash? Yes!!!
Better than shell scripts? Worse?
Better than any of the million other tools? Maybe...
Better than hand-rolling something? Not so sure now...
Problem: Is your automation efficient?
Now we can spin up & setup servers with a few simple commands. But it wasslow... Base+nginx+SSL+Python+app often took ~2h. We hated it and weavoided it.
Solution? Create a base image
First, we tried Packer (http://packer.io/) - but it had annoying bugs; was very difficultto debug
Second attempt: plain Ansible - worked much better
Maybe there are better tools to do these things?
We haven't even tried any of the other popular solutions: Chef, Puppet,Fabric, Atlas/Terraform, etc.
Some take-aways
No automation is the worst.
Slow automation is no better than no automation.
Buggy automation is even worse!
Solutions can sometimes create more problems.
Research your tools, pick stuff that works for your needs!
Problem: So how exactly do you deploy these"kittens"?
Our company has very dynamic and growing needs
Remember rule #1: automate the repetitive, boring, daunting stuff
Sysop team spent days writing elaborate Ansible playbooks to set up serversfor projects
But wait! Rule #2: can't automate something? Delegate it!
First attempt: let developers write Ansible playbooks...
Ansible has a pretty steep learning curve and some serious limitations
Difficulties micro-managing access
Difficulties achieving parity between local dev, vagrant, CI, stage, live...
Result: playbooks full of ugly hacks
Second attempt: Docker
Docker: some seriously magical stuffhappened
Docker puts apps into portable "containers"
Many difficult problems become easy
Developers are (mostly) in charge of their Dockerfiles
Developers no longer blocked by sysadmins
Builds are 100% automated and 100% reproducible
Much easier to review, audit, understand
Output is a fixed image; if it runs well in dev, stage, it will (probably) run wellin prod
Our deploy playbooks for Ansible are basically copy-paste & tweak; workunderway to unify them
We've created a set of base images that make it easy to follow our internalconventions - people stick to conventions more often
Docker: but not everything is so rosy!
Another tool to learn for the (always busy) developers.
Difference between "build time" and "run time" greatly emphasized, for thebetter and for the worse.
# XXX broken? RUN service nginx start RUN curl localhost
Many difficult problems stay difficult, or become even harder.
Not a silver bullet: we've dockerized most of our apps, but Ansible stillmanages the raw computing infrastructure.
(You can pry Ansible from my cold, dead hands!)
(But I'm still writing my own configuration management tool, as a sideproject.)
More conclusions?
Choose tools that empower your team-mates!
Different tools for different types of tasks
Automation: For IT staff
Often burdened with stuff that's not always easily automated:
Purchases, finance
Onboarding, helpdesk, troubleshooting
Setup, wiring, "hands" in the server room
Often burdened with daunting, scriptable tasks, but no time to develop theautomation
At our company, sysadmins and "IT guys" often work together closely
Some classical problemsHave hundreds of groups on G Suite. Review and delete empty ones.
Review group membership; create list of users from outside the domain.
Update everyone's email signatures.
Ensure there are no rogue .mp3's on anyone's Google Drive.
Create a project on JIRA with all the standard workflows, templates, etc.
...all of that, regularly.
Solutions?
Write some simple scripts!
Our in-house tool: https://bitbucket.org/u9/gadmin
E.g. to delete empty groups on G Suite:
$ gadmin groups count example.com [email protected] 11 [email protected] 0 [email protected] 0 [email protected] 4 [email protected] 0 $ gadmin groups count example.com | awk '$2=="0" {print $1}' \ | xargs -r -n1 gadmin groups delete
Very much WIP, it may eat your laundry ;)
Solutions?
Write some more sophisticated scripts!
We have in-house tools for:
Device tracking for testers
JIRA project setup
Processing invoices
URL shortening
And more...
Solutions?
Use a third-party service!
We use BetterCloud - https://www.bettercloud.com/
Automates stuff like email signatures, Drive sharing policies, userprovisioning, and a lot more...
Automation: For developers: Hubot
Chatbot; built internally at Github, open-sourced: https://hubot.github.com/
Plenty of community-maintained plugins
Easy to roll your own: JS, Coffee...
We use it internally for a whole bunch of small stuff:
Standup alarms
hubot: @here it's 11:00 AM, standup time! https://hangups.example.com/kittens-standup
Trigger a deployment
me: hubot deploy kittens 0.2.17 to dev
hubot: OK! Running:
hubot:ansible-playbook deploy-kittens.yml -e version=0.2.17 -e env=dev
Automation: For developers: Hubot
...and for fulfilling our random gif needs
me: hubot pug me
hubot: https://media.tumblr.com/tumblr_lisv04akTl1qb08qmo1_500.jpg
Automation: For developers: makeOld, venerable tool - 1976, probably older than you
Surprisingly versatile
(If your problem can be expressed as producing an output file from abunch of input files!)
Build your project (obviously)
Maintain .gitignore via https://www.gitignore.io/
Maintain your scripts & dotfiles across machines (make + git)
These slides were made with make ;)
%.html: %.md @pandoc --self-contained -f markdown -t slidy $< -o $@
make practical-automation.html
Automation: For everyone!
Creating sophisticated automation requires coding skills, understanding ofthe problem domain, commitment to maintenance...
But not all tools need to be THIS sophisticated!
If enough people have the same problem...
Automation: For mere mortals: GUI tools,Automation-as-a-Service...
Plenty of options, domain-specific or more general:
AutoHotKey - https://www.autohotkey.com/ - for Windows users
BetterCloud - https://www.bettercloud.com/ - G Suite and more
If This Then That - https://ifttt.com/
Zapier - https://zapier.com/
Various Slack bots; off-the-shelf Hubot plugins
Questions? Comments?Any tools YOU would recommend?
Success stories?
Hate mail?
I will answer everything :)
cat /dev/audience \ | while read question do echo 42 > /dev/audience done
Thanks!
