Practicing Continuous Deployment

download Practicing Continuous Deployment

of 38

  • date post

    15-Jan-2015
  • Category

    Technology

  • view

    2.773
  • download

    0

Embed Size (px)

description

 

Transcript of Practicing Continuous Deployment

  • 1. DISQUS Practicing Continuous DeploymentDavid Cramer @zeegSaturday, March 10, 12

2. Shipping new code as soon as its readySaturday, March 10, 12 3. Continuous Deployment # Update the site every 5 minutes */5 * * * * cd /www/example.com&& git pull&& service apache restartSaturday, March 10, 12 4. Saturday, March 10, 12 5. When its readySaturday, March 10, 12 6. When is it ready?-Reviewed by peers-Passes automated tests-Some level of QASaturday, March 10, 12 7. Focus on Stability and IterationSaturday, March 10, 12 8. WorkowReviewCommit Integration Failed BuildDeploy ReportingRollbackSaturday, March 10, 12 9. The Good- Develop featuresincrementally- Release frequently- Smaller doses of QAThe Bad- Culture Shock- Stability depends ontest coverage- Initial time investmentSaturday, March 10, 12 10. Keep Development SimpleSaturday, March 10, 12 11. Development-Automate testing of complicated processes and architecture-Simple can be better than complete - Especially for local development-python setup.py {develop,test}-Puppet, Chef, Buildout, Fabric, etc.Saturday, March 10, 12 12. Production StagingPostgreSQLPostgreSQLMemcacheMemcacheRedis RedisSolrSolrApacheApacheNginx NginxRabbitMQRabbitMQ (and 100 other painful-to-congure services)CI Server MacbookPostgreSQLPostgreSQLMemcacheApacheRedis MemcacheSolrRedisApacheSolrNginx NginxRabbitMQRabbitMQSaturday, March 10, 12 13. Bootstrapping Local-Simplify local setup - git clone dcramer@disqus:disqus.git - make - python manage.py runserver-Need to test dependancies? - virtualbox + vagrant upSaturday, March 10, 12 14. Progressive Rollout We actively use early versions of features before public releaseSaturday, March 10, 12 15. Deploy features to portions of a user base at atime to ensure smooth, measurable releaseshttps://github.com/disqus/gargoyleSaturday, March 10, 12 16. Iterate quickly by hiding featuresEarly adopters are free QA from gargoyle import gargoyle def my_view(request): if gargoyle.is_active(awesome, request): return new happy version :D else: return old sad version :(Saturday, March 10, 12 17. SWITCHES = { # enable my_feature for 50% my_feature: range(0, 50), } def is_active(switch): try:pct_range = SWITCHES[switch] except KeyError:return False ip_hash = sum([int(x) for xin ip_address.split(.)]) return (ip_hash % 100 in pct_range)Saturday, March 10, 12 18. Review ALL the Commitsphabricator.orgSaturday, March 10, 12 19. Saturday, March 10, 12 20. Saturday, March 10, 12 21. Saturday, March 10, 12 22. Integration (or as we like to call it)Saturday, March 10, 12 23. Saturday, March 10, 12 24. Integration Requirements-Developers must know when theyve broken something - IRC, Email, IM-Support proper reporting - XUnit, Pylint, Coverage.py-Painless setup - apt-get install jenkins * https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+UbuntuSaturday, March 10, 12 25. Shortcomings-False positives - Reporting isnt accurate - Services fail - Bad Tests-Test coverage - Regressions on untested code-Feedback delay - Integration tests vs Unit testsSaturday, March 10, 12 26. Fixing False Positives-Re-run tests several times on a failure-Report continually failing tests-Replace external service tests with a functional test suiteSaturday, March 10, 12 27. Maintaining Coverage-Raise awareness with reporting - Fail/alert when coverage drops on a build-Commit tests with code - Coverage against commit diff for untested regressions-Utilize code reviewSaturday, March 10, 12 28. Speeding Up Tests-Write true unit tests - vs slower integration tests-Mock external services-Distributed and parallel testing - Matrix buildsSaturday, March 10, 12 29. ReportingSaturday, March 10, 12 30. Why is mongodb-1 down? Its down? Must have crashed againSaturday, March 10, 12 31. Meaningful Metrics-Rate of traffic (not just hits!) - Business vs system-Response time (database, web)-Exceptions-Social media - TwitterSaturday, March 10, 12 32. Graphite (Trafficgraphite.wikidot.comacross a cluster of servers)Saturday, March 10, 12 33. Sentry sentry.readthedocs.orgSaturday, March 10, 12 34. Wrap UpSaturday, March 10, 12 35. Getting Started-Package your app-Value code review-Ease deployment; fast rollbacks-Setup automated tests-Gather some easy metricsSaturday, March 10, 12 36. Going Further-Build an immune system - Automate deploys, rollbacks (maybe)-Adjust to your culture - There is no right way-SOA == great successSaturday, March 10, 12 37. DISQUSQuestions?psst, were hiringdisqus.com/jobsSaturday, March 10, 12 38. References-Gargoyle (feature switches) https://github.com/disqus/gargoyle-Sentry (log aggregation) https://github.com/dcramer/sentry-Jenkins CI (continuous integration) http://jenkins-ci.org/-Phabricator (code reviews, bug tracking) https://phabricator.org-Graphite (metrics) http://graphite.wikidot.com/code.disqus.comSaturday, March 10, 12