Zero Downtime Deployment with AnsibleDevOpsCon Berlin 2016

@steinim

stein.inge.morisbak@BEKK.no

OPEN

Slideshttp://steinim.github.io/slides/devopscon/zero-downtime-ansible

Source codegit@github.com:steinim/zero-downtime-ansible.git

Full tutorialhttp://steinim.github.io/slides/devopscon/zero-downtime-

ansible/tutorial.html

What's a provisioning framework?Automated setup of serversConfiguration as codeNot immutable (in itself)

ExamplesCreate usersInstall softwareGenerate and manipulate config filesStart/stop/restart processesSet up dependencies between operations

Describe what to do (imperative)#!/bin/bash if $( command -v vim >/dev/null 2>&1 ); then echo "vim is already installed." else apt-get install vim fi if $( grep -Fxq "filetype indent off" /etc/vim/vimrc ); then echo "set filetype indent off is already in /etc/vim/vimrc." else echo "filetype indent off" >> /etc/vim/vimrc # TODO: Do not continue if this fails. fi # TODO: Rollback if something fails.

Describe state (declarative)- name: ensure installed vim apt: pkg=vim state=installed - name: set filetype indent off for vim lineinfile: dest=/etc/vim/vimrc line='filetype indent off' state=present

ProsIn source control.Self documenting (it's code!).Refactoring.Less differences between environments.Deterministic.Prevents manual steps.Fast and easy to configure up a new environment.Easier to test server setup.

AnsibleSSH-basedClient only (no server)YAML configurationPush (and pull)Supports more than setup and provisioning:

Application deploymentRemote command execution

Bring up the boxesvagrant up

Layout (convention over configuration)├── ansible.cfg ├── hosts ├── site.yml ├── group_vars │   └── <group name> ├── host_vars │   └── <host name> ├── roles │   ├── <role> │   │   ├── files │   │   └── <file> │   │   └── templates │   │   └── <template>.j2 │   │   ├── handlers │   │   │   └── main.yml │   │   ├── tasks │   │   │   └── main.yml │   │   ├── vars │   │   │   └── main.yml

Play!ansible-playbook site.yml

FactsAnsible by default gathers “facts” about the machines undermanagement.These facts can be accessed in Playbooks and in templates.ansible -m setup app1.local

The taskAn app user 'devops', with:

Home directory: /home/devopsssh-key

A PostgresSQL database.Nginx as a reverse proxy.An init script installed as a service.Deploy an application that uses the provisioned infrastructure.

Help!http://docs.ansible.com/list_of_all_modules.html

Task1: Install and configure softwareModify roles/common/tasks/apt.yml.Install Vim.Insert the line 'filetype indent off' in /etc/vim/vimrc

Help:

http://docs.ansible.com/apt_module.htmlhttp://docs.ansible.com/lineinfile_module.html

Task1: Solution- name: Ensure installed vim apt: pkg=vim state=present update_cache=no tags: - vim - name: Set filetype indent off lineinfile: dest=/etc/vim/vimrc line='filetype indent off' state=present tags: - vim

Run it!ansible-playbook site.yml --tags vim

ProTip: Use '--tags', '--skip-tags', '--limit' and/or 'gather_facts: False'to reduce execution time.

ProgressInstalled softwareManipulated files

VariablesUse variables! → Infrastructure as data.Where should variables be defined? Ansible has very many options.

Inventory (./hosts)group_vars and host_varsPlaybook (site.yml)Facts (local or server)Command line (as arguments)

Access variables from playbooks: "{{ variable }}"

http://docs.ansible.com/playbooks_variables.html

Task2: Create an application userCreate roles/users/tasks/main.ymlHome directory: /home/devopsssh-keyUse variables! (group_vars)

Help: (create a directory)

(.ssh/authorized_keys)

http://docs.ansible.com/group_module.htmlhttp://docs.ansible.com/user_module.htmlhttp://docs.ansible.com/file_module.htmlhttp://docs.ansible.com/lineinfile_module.htmlhttp://docs.ansible.com/playbooks_best_practices.html#group-and-host-variables

Define some variables# group_vars/appservers

user: devops group: devops user_name: "Devops app user"

Task2: Solution# roles/users/tasks/main.yml

- name: Create group group: name={{ group }} state=present tags: - users - name: Create user user: name={{ user }} comment="{{ user_name }}" group={{ group }} createhome=yes home=/home/{{ user }} shell=/bin/bash state=present tags: - users - name: Ensure ssh directory exists for user

Run it!ansible-playbook site.yml --limit appservers --skip-tags apt,vim,java

Try it!ssh devops@app1.local

ProgressInstalled softwareManipulated filesCreated a user and set up a ssh-key

Task3: Install and configure PostgreSQLroles/postgresql ├── files │   └── postgresql.conf ├── handlers │   └── main.yml ├── tasks │   ├── main.yml │   └── ... └── templates └── pg_hba.conf.j2

Use variables (group_vars/dbservers).

Template: roles/postgresql/templates/pg_hba.conf.j2

Use handler to restart postgresql upon notification

Help: (pg_hba.conf.j2)

http://docs.ansible.com/template_module.htmlhttp://docs.ansible.com/postgresql_user_module.htmlhttp://docs.ansible.com/postgresql_db_module.htmlhttp://docs.ansible.com/playbooks_intro.html#handlers-running-operations-on-change

Define some variables# group_vars/dbservers

postgresql: version: 9.4 repo: 'deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main' user: postgres group: postgres data_dir: /var/lib/pgsql/data address: 192.168.101.0/24 db: name: devops user: devops password: devops123

Create a template# roles/postgresql/templates/pg_hba.conf.j2

# TYPE DATABASE USER ADDRESS METHOD local all all trust host {{ db.name }} {{ db.user }} {{ postgresql.address }} md5

Create a handler# roles/postgresql/handlers/main.yml

- name: Restart postgresql service: name=postgresql state=restarted

Task3: Solution# roles/postgresql/tasks/postgresql.yml

- name: Install Postgresql and dependencies apt: pkg={{ item }} state=installed update_cache=yes with_items: - postgresql-{{ postgresql.version }} - python-psycopg2 tags: - pg_install - name: Install postgresql.conf copy: src=postgresql.conf dest=/etc/postgresql/{{ postgresql.version }}/main/postgresql.conf owner={{ postgresql.user }} group={{ postgresql.group }} notify: - Restart postgresql tags: - pg_install

Run it!ansible-playbook site.yml --limit dbservers --tags pg_install

Try it!$ vagrant ssh db vagrant@db:~$ psql -d devops -U devops -W devops=> \q

ProgressInstalled softwareManipulated filesCreated a user and set up a ssh-keyInstalled and configured a database and a db user

Task4: Deploy!roles/app ├── files │   └── init.sh ├── tasks │   └── main.yml └── templates └── config.properties.j2

NB! Use variables (./hosts).

Set 'serial: 1' for appservers in the playbook (site.yml).

Help:

http://docs.ansible.com/service_module.html

Browse to

Run it!ansible-playbook site.yml --limit appservers --tags deploy

Try it!http://app1.local:1234/

What just happened?/home/devops ├── config.properties ├── current -> /home/devops/devops_1416228023.jar ├── previous -> /home/devops/devops_1416221573.jar ├── devops_1416221573.jar ├── devops_1416228023.jar └── logs ├── stderr.log └── stdout.log

/etc/init.d └── devops

ProgressInstalled softwareManipulated filesCreated a user and set up a ssh-keyInstalled and configured a database and a db userDeployed an application to two appservers and enabled it as aservice

Task5: Deploy databaseroles/db ├── files │   └── migrate_db.sql └── tasks └── main.yml

Help: http://docs.ansible.com/command_module.html

Task5: Solution- name: Copy db migration script copy: src=migrate_db.sql dest=/tmp/migrate_db.sql become_user: postgres tags: - deploy - name: Run db migration script command: psql -d {{ db.name }} -q -f /tmp/migrate_db.sql become_user: postgres tags: - deploy

Browse to

Run it!ansible-playbook site.yml --limit dbservers --tags deploy

Try it!$ vagrant ssh db vagrant@db:~$ psql -d devops -U devops -W devops=> \dt devops=> select * from hello; devops=> \q

http://app1.local:1234/

ProgressInstalled softwareManipulated filesCreated a user and set up a ssh-keyInstalled and configured a database and a db userDeployed an application to two appservers and enabled it as aserviceMigrated the database schema and fetched data from it through theapplication

Task6: Set up proxyroles/nginx ├── handlers │   └── main.yml ├── tasks │   ├── config_nginx.yml │   ├── install_nginx.yml │   └── main.yml └── templates └── devops.conf.j2

Help:

http://wsgiarea.pocoo.org/jinja/docs/loops.html

Task6: Solutionupstream backend { {% for appserver in groups.appservers %} server {{ appserver }}:{{ app_port }} fail_timeout=1s; {% endfor %} } server { listen 80; server_name _; access_log /var/log/nginx/devops-access.log; location / { proxy_pass http://backend; proxy_redirect off; proxy_next_upstream error timeout invalid_header http_500 http_502; proxy_connect_timeout 2; } }

Browse to # refresh me many times

Run it!ansible-playbook site.yml --limit proxies --tags nginx

Try it!http://proxy.local/

ProgressInstalled softwareManipulated filesCreated a user and set up a ssh-keyInstalled and configured a database and a db userDeployed an application to two appservers and enabled it as aserviceMigrated the database schema and fetched data from it through theapplicationSet up a reverse proxy for automatic failover between the twoappservers

The Expand/Contract pattern

Expand ContractAdd tablesAdd columnsTweak indexes

Remove tablesRemove columnsRemove/add constraints

Play time :-)Suggestions:

Change database table name from HELLO to MESSAGES anddeploy a new version without downtime.Implement automated rollback.

September 5th and 6th 2016

Thank you!

@steinim

stein.inge.morisbak@BEKK.no