Continuous Deployment at Disqus (Pylons Minicon)
-
Upload
zeeg -
Category
Technology
-
view
5.046 -
download
1
Transcript of Continuous Deployment at Disqus (Pylons Minicon)
![Page 1: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/1.jpg)
DISQUSContinuous Deployment Everything
David Cramer@zeeg
Friday, April 29, 2011
![Page 2: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/2.jpg)
Disclaimer: We don’t use Pylons
Friday, April 29, 2011
![Page 3: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/3.jpg)
Shipping new code as soon as it’s ready
(It’s really just super awesome buildbots)
Continuous Deployment
Friday, April 29, 2011
![Page 4: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/4.jpg)
Workflow
1. Developer commits code
2.CI server runs tests automatically
2.1.Build passes, code deploys
2.2.Build fails, block deploy
3.Developer tests feature on production
Friday, April 29, 2011
![Page 5: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/5.jpg)
Pros
• Develop features incrementally
• Release frequently
• Less QA! (maybe)
Cons
• Culture Shock• Stability depends on
test coverage
• Initial time investment
We mostly just care about iteration and stability
Friday, April 29, 2011
![Page 6: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/6.jpg)
Painless Development
Friday, April 29, 2011
![Page 7: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/7.jpg)
Development
• Production > Staging > CI > Dev
• Automate testing of complicated processes and architecture
• Simple is arguably better than complete
• Especially for local development
• python setup.py {develop,test}• Use Puppet/Chef or build a simple
bootstrap.{py,sh}
Friday, April 29, 2011
![Page 8: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/8.jpg)
Production Staging
CI Server Macbook
• PostgreSQL• Memcache• Redis• Solr• Apache• Nginx• RabbitMQ
• PostgreSQL• Memcache• Redis• Solr• Apache • Nginx• RabbitMQ
• Memcache• PostgreSQL• Redis• Solr• Apache• Nginx• RabbitMQ
• PostgreSQL• Apache• Memcache• Redis• Solr• Nginx• RabbitMQ
Friday, April 29, 2011
![Page 9: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/9.jpg)
Bootstrapping Local
• Simplify local setup
• git clone dcramer@disqus:disqus.git• python setup.py develop• python manage.py runserver
• Need to test dependancies?
• virtualbox + vagrant up
Friday, April 29, 2011
![Page 10: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/10.jpg)
“Under Construction”
from gargoyle import gargoyle
def my_view(request): if gargoyle.is_active('awesome', request): return 'new happy version :D' else: return 'old sad version :('
• Iterate quickly by hiding features
• Early adopters are free QA
Friday, April 29, 2011
![Page 11: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/11.jpg)
Gargoyle
Being users of our product, we actively use early versions of features before public release
Deploy features to portions of a user base at a time to ensure smooth, measurable releases
Friday, April 29, 2011
![Page 12: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/12.jpg)
Integration
(or as we like to call it)
Friday, April 29, 2011
![Page 13: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/13.jpg)
Integration is Required
Deploy only when things wont break
Friday, April 29, 2011
![Page 14: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/14.jpg)
Reporting is Critical
Friday, April 29, 2011
![Page 15: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/15.jpg)
CI Requirements
✓ Developers must know when they’ve broken something
✓ IRC, Email, IM
✓ Support proper reporting
✓ XUnit, Pylint, Coverage.py
✓ Painless setup
✓ apt-get install jenkins
Friday, April 29, 2011
![Page 16: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/16.jpg)
Shortcomings
• False positives lower awareness
• Reporting isn't accurate
• Services fail
• Bad Tests
• Not enough code coverage
• Regressions on untested code
• Test suite takes too long
• Integration tests vs Unit tests
Friday, April 29, 2011
![Page 17: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/17.jpg)
Fixing False Positives
• Re-run tests several times on a failure
• Report continually failing tests
• Fix continually failing tests
• Rely less on 3rd parties
• Mock/Dingus
Friday, April 29, 2011
![Page 18: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/18.jpg)
Maintaining Coverage
• Raise awareness with reporting
• Fail/alert when coverage drops on a build
• Commit tests with code
• Drive it into your culture
Friday, April 29, 2011
![Page 19: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/19.jpg)
Speeding Up Tests
• Write true unit tests
• vs slower integration tests
• Mock 3rd party APIs
• Distributed and parallel testing
• http://github.com/disqus/mule
Friday, April 29, 2011
![Page 20: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/20.jpg)
Mule
• Unstable (we’re working on it)
• Mostly Django right now
• Generic interfaces for unittest2• Works with multi-processing and Celery
• More complex than normal Celery usage
• Full XUnit integration
• Simple workflow
• mule test --runner="python manage.py mule --worker $TEST"
Friday, April 29, 2011
![Page 21: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/21.jpg)
Deploy (finally)
Friday, April 29, 2011
![Page 22: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/22.jpg)
How DISQUS Does It
• Incremental deploy with Fabric
• Drop server from pool
• Pull in requirements on each server
• Isolated virtualenv’s built on each server
• Push server back online
Friday, April 29, 2011
![Page 23: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/23.jpg)
Challenges
• PyPi works on server A, but not B
• Scale, or lack of
• CPU cost per server
• Schema changes, data model changes
• Backwards compatibility
Friday, April 29, 2011
![Page 24: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/24.jpg)
PyPi is Down
• http://github.com/disqus/chishop
Friday, April 29, 2011
![Page 25: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/25.jpg)
Help, we have 100 servers!
• Incremental (ours) vs Fanout
• Push vs Pull
• Twitter uses BitTorrent
• Isolation vs Packaging (Complexity)
Friday, April 29, 2011
![Page 26: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/26.jpg)
1. Add column (NULLable)2. Add app code to fill column3. Deploy4. Backfill column5. Add app code to read column6. Deploy
SQL Schema Changes
Cached Data Changes
• Have a global version number• Have a data model cache version
• maybe md5(cls.__dict__)?
Friday, April 29, 2011
![Page 28: Continuous Deployment at Disqus (Pylons Minicon)](https://reader033.fdocuments.in/reader033/viewer/2022052310/554bb403b4c9053a298b46ca/html5/thumbnails/28.jpg)
References
• Mule (distributed test runner)http://github.com/disqus/mule
• Gargoyle (feature switches)https://github.com/disqus/gargoyle
• Jenkins CIhttp://jenkins-ci.org/
code.disqus.com
Friday, April 29, 2011