DISQUSPracticing Continuous Deployment

David Cramer@zeeg

Saturday, March 10, 12

Shipping new code as soon as it’s ready

Saturday, March 10, 12

Continuous Deployment

# Update the site every 5 minutes*/5 * * * * cd /www/example.com \ && git pull \ && service apache restart

Saturday, March 10, 12

Saturday, March 10, 12

When it’s ready

Saturday, March 10, 12

When is it ready?

- Reviewed by peers

- Passes automated tests

- Some level of QA

Saturday, March 10, 12

Focus on Stability and Iteration

Saturday, March 10, 12

Workflow

Review

Integration

Deploy

Failed Build

Reporting

Rollback

Commit

Saturday, March 10, 12

- Develop features incrementally

- Release frequently

- Smaller doses of QA

- Culture Shock

- Stability depends on test coverage

- Initial time investment

The Good

The Bad

Saturday, March 10, 12

Keep Development Simple

Saturday, March 10, 12

Development

- Automate testing of complicated processes and architecture

- Simple can be better than complete

- Especially for local development

- python setup.py {develop,test}- Puppet, Chef, Buildout, Fabric, etc.

Saturday, March 10, 12

Production Staging

CI Server Macbook

PostgreSQLMemcacheRedisSolrApacheNginxRabbitMQ

PostgreSQLMemcacheRedisSolrApache NginxRabbitMQ

PostgreSQLMemcacheRedisSolrApacheNginxRabbitMQ

PostgreSQLApacheMemcacheRedisSolrNginxRabbitMQ

(and 100 other painful-to-configure services)

Saturday, March 10, 12

Bootstrapping Local

- Simplify local setup

- git clone dcramer@disqus:disqus.git- make- python manage.py runserver

- Need to test dependancies?

- virtualbox + vagrant up

Saturday, March 10, 12

Progressive Rollout

We actively use early versions of features before public release

Saturday, March 10, 12

Deploy features to portions of a user base at a time to ensure smooth, measurable releases

https://github.com/disqus/gargoyle

Saturday, March 10, 12

from gargoyle import gargoyle

def my_view(request): if gargoyle.is_active('awesome', request): return 'new happy version :D' else: return 'old sad version :('

• Iterate quickly by hiding features

• Early adopters are free QA

Saturday, March 10, 12

SWITCHES = { # enable my_feature for 50% 'my_feature': range(0, 50),}

def is_active(switch): try: pct_range = SWITCHES[switch] except KeyError: return False

ip_hash = sum([int(x) for x in ip_address.split('.')])

return (ip_hash % 100 in pct_range)

Saturday, March 10, 12

Review ALL the Commits

phabricator.org

Saturday, March 10, 12

Saturday, March 10, 12

Saturday, March 10, 12

Saturday, March 10, 12

Integration

(or as we like to call it)

Saturday, March 10, 12

Saturday, March 10, 12

Integration Requirements

- Developers must know when they’ve broken something

- IRC, Email, IM

- Support proper reporting

- XUnit, Pylint, Coverage.py

- Painless setup

- apt-get install jenkins *

https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu

Saturday, March 10, 12

Shortcomings

- False positives- Reporting isn't accurate

- Services fail

- Bad Tests

- Test coverage- Regressions on untested code

- Feedback delay- Integration tests vs Unit tests

Saturday, March 10, 12

Fixing False Positives

- Re-run tests several times on a failure

- Report continually failing tests

- Replace external service tests with a functional test suite

Saturday, March 10, 12

Maintaining Coverage

- Raise awareness with reporting

- Fail/alert when coverage drops on a build

- Commit tests with code

- Coverage against commit di! for untested regressions

- Utilize code review

Saturday, March 10, 12

Speeding Up Tests

- Write true unit tests

- vs slower integration tests

- Mock external services

- Distributed and parallel testing

- Matrix builds

Saturday, March 10, 12

Reporting

Saturday, March 10, 12

<You> Why is mongodb-1 down?

<Ops> It’s down? Must have crashed again

Saturday, March 10, 12

Meaningful Metrics

- Rate of tra"c (not just hits!)

- Business vs system

- Response time (database, web)

- Exceptions

- Social media

- Twitter

Saturday, March 10, 12

Graphite

(Tra!c across a cluster of servers)graphite.wikidot.com

Saturday, March 10, 12

sentry.readthedocs.org

Sentry

Saturday, March 10, 12

Wrap Up

Saturday, March 10, 12

Getting Started

- Package your app

- Value code review

- Ease deployment; fast rollbacks- Setup automated tests

- Gather some easy metrics

Saturday, March 10, 12

Going Further

- Build an immune system

- Automate deploys, rollbacks (maybe)

- Adjust to your culture

- There is no “right way”

- SOA == great success

Saturday, March 10, 12

DISQUSQuestions?

psst, we’re hiringdisqus.com/jobs

Saturday, March 10, 12

References

- Gargoyle (feature switches)https://github.com/disqus/gargoyle

- Sentry (log aggregation)https://github.com/dcramer/sentry

- Jenkins CI (continuous integration)http://jenkins-ci.org/

- Phabricator (code reviews, bug tracking)https://phabricator.org

- Graphite (metrics)http://graphite.wikidot.com/

code.disqus.com

Saturday, March 10, 12