Pragmatic steps on the path to continuous delivery

Zero to Hero

@geoffnettaglich

Continuous Delivery : Theory

10 have idea20 implement idea

30 measure effectiveness

40 refine idea

50 GOTO 10

60 PROFIT

build / deploy / feedback

automated

Continuous Delivery : Reality

● Everyone is doing it (apparently)○ Amazon, Etsy, Facebook, Netflix

■ N deploys per day / min (where N > 0)○ Great idea (in theory)

● What about me?○ Existing codebase (smells and all)○ Existing systems & environments (no access)○ Existing team and politics (no idea)

inertia

Continuous Delivery : Practice

● Focus on the journey○ it will be ongoing anyway○ simplify / automate

Suck less tomorrow!

● Crawl / Walk / Run○ Find a place to start

■ most painful■ simplest entry point

Build : Theory

● Dependencies : what we needlib of Jars / Maven / Ivy

● Artifacts : what we producetgz / jar / war

● Reproducible : scriptedAnt / Maven / Gradle

● AutomatableJenkins

Build : Reality

● GOOD○ Using ant + ivy + sh + junit

● NOT SO GOOD○ Issues with build dependencies○ No output artifact○ No easy way to create DB○ Only worked on developers machine○ Took a day or two to get new team members working

Build : Practice

● Tidy up existing ant build (based on intent)○ clean / compile / test / package○ create / bootstrap DB locally

● Bundle build properties● Run app locally● Generate reports

○ Unit tests (JUnit)○ Static analysis (FindBugs et al)

checkout / build / run

Deploy : Theory● Push button / Zero downtime

○ Not always easy: Java != rails != php/apache

● Deploy from developers machine is FAIL○ Shared env or jump box○ Then automate (via Jenkins)

● What version is running where○ Build page○ All servers up to date with latest version

Deploy : Reality

● GOOD○ Scripted (somewhat)

● NOT SO GOOD○ Only from developers machine [FAIL!]○ Ignored errors (FULL STEAM AHEAD)○ Rsync of build outcome○ Manual stop / start ○ Customers may see errors during rollout

Deploy : Practice

● bash script (Capistrano or similar)○ stop / upload / unpack / restart○ similar structure to capistrano on filesystem

● Verify○ Build page with stats (bundled in artifact)

● DB changes○ Backwards compatible changes (process)○ Liquibase is great

● Automated (via Jenkins of course)

Run : Theory● Elastic scaling

○ Dynamic provisioning○ Automated service management

● Automated monitoring and alerting○ Sensu / Nagios / ganglia○ LogStash / GrayLog2

● Dashboards for EVERYTHING!

Run : Reality● ssh to servers

○ to start stop services○ monitor perf via top and ps○ grep and tail -f of log files

● Only ‘qualified’ admins allowed● No business metrics (manual reports)● No visibility● RESTART

WARNING!

Prone to human ERORS

● New Relic○ servers for free○ app with paid ○ great insight

● Centralized logging○ rsyslog:unification/collection○ papertrail:aggregation & display

Run : Practice

● Health Metrics (in app)○ home rolled exposed via web○ Metrics by CodaHale / Pingdom

● Biz Metrics○ DucksBoard

■ simple funnel■ attrition and activity

● Monit○ alert and restart (if needed)

Run : Practice

● Scale – vertically or horizontally● automatable, reproducible

○ Green / Blue deploys FTW● Chef / Puppet / Ansible ... just pick one

● If a server fails, can you rebuild it?● If an environment fails, can you rebuild it?

Environments : Theory

‘Works on my machine’

Environments : Reality

● Private cloud hosting● Limited upgrade ability● Manual updates● Uncertain/misunderstood foundation● SSH and rsync to update and deploy

○ overwrote existing codebase

You won’t always know what you inherit!

Environments : Practice● Picked Chef (Vi vs Emacs …)

○ Roles for baseline / Web / App / DB nodes○ Environments for DEV / STG / PRD

● Vagrant for testing recipes locally○ config checked into app project

● Automate (via Jenkins of course)○ validate cookbooks via FoodCritic ○ upload to hosted chef○ rollout / reprovision via bash / ssh

Environments : Practice● DB backup / restore

○ do it and test it regularly○ scripted○ hire an expert

Delegate to others

● Mandrill for outgoing SMTP● DNSimple for managed DNS● Pingdom for simple uptime

○ custom health check endpoint

Rackspace Cloud

Hosted chef

Managed by Jenkins○ Re-Create dev / staging / production○ deploy staging / production

Monitored via○ Pingdom○ New Relic○ PaperTrail app

Rolled out to iOS app dev TOO!○ Xcode CLI / OCUnit / Testflight via Jenkins

Environment : Current State

OH: I deployed to production, nobody noticed and nothing went wrong

FTW!