Tech Talk: Dont Forget Operations in Your Continuous Delivery Practice
Continuous delivery in practice (public)
-
Upload
tzach-zohar -
Category
Software
-
view
645 -
download
0
description
Transcript of Continuous delivery in practice (public)
![Page 1: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/1.jpg)
Continuous Delivery In Practice
Lessons from Kenshoo’s RTB project
![Page 2: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/2.jpg)
Who, What, Where
Tzach Zohar:● System Architect● [email protected]
Kenshoo: ● Founded 2006● Online Marketing Technology● ~450 employees● 12 World Wide locations
![Page 3: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/3.jpg)
Continuous Delivery: Why bother?
Faster development, delivery and feedback makes our clients and us happier.
The “us” alone is worth it.
![Page 4: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/4.jpg)
Continuous Delivery: Why bother?
Better suited productResponsiveness
Less wasteHigher quality
![Page 5: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/5.jpg)
Background: RTB Project
● ~1.5 years ● ~4 developers● ~Dozens of paying clients● ~50 servers (AWS)● ~1.5M requests per minute● ~7ms average response time● ~99.9% availability● ~5-10 deployments / week
![Page 6: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/6.jpg)
How?
![Page 7: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/7.jpg)
1.The Obvious
● Single branch● Full, Fast, Reliable coverage● Full deployment automation● Fast feedback● ABCD - Always Be Continuously
Deploying
![Page 8: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/8.jpg)
● Unit: complete functional coverage● Integration: with external systems - thin!● Behavioral: we use Cucumber● Staging: verify actual server upgrade
2. Four-Layer Test Suite
![Page 9: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/9.jpg)
2. Four-Layer Test Suite
![Page 10: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/10.jpg)
Staging: verify compatibility of new build with other components’ production builds
2. Four-Layer Test Suite
![Page 11: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/11.jpg)
3. Keep Builds Stable
Do not overlook a test that “sometimes fails”, trusting build status is crucial
![Page 12: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/12.jpg)
3. Keep Builds Stable
Be Suspicious of:● Random data tests● Asynchronous tests● Integration tests
![Page 13: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/13.jpg)
4. Master Is Always
Shippable
On every commit? Not Quite:Use GitHub Pull Requests
“Merge” == Build and Deploy
credit: [email protected]
![Page 14: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/14.jpg)
5. Rigorous Code Reviews
● Remember “merge” means “deploy”!● Insist on proper coverage● Insist on code cleanliness● Insist on consistent design● Insist!
![Page 15: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/15.jpg)
5. Rigorous Code Reviews
https://github.com/tzachz/github-comment-counter
![Page 16: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/16.jpg)
6. Real-Time Feedback
Detect issues immediately and visually
![Page 17: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/17.jpg)
7. Keep Upgrade in Mind (1)
Use the “Parallel Change” pattern when changing cross-node APIs / Data
1.Write: oldRead: both
2.Write: new Read: both
3.Write: new Read: new
deploy deploy
![Page 18: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/18.jpg)
8. Keep Upgrade in Mind (2)
Verify backward compatibility in tests
![Page 19: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/19.jpg)
9. A/B Testing
Apply new features to a limited user-group Measure business results per-group
(Not by branching)
![Page 20: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/20.jpg)
10. Own It
Constantly check buildsConstantly collect feedbackConstantly check monitorsAnswer the phone at 3am
![Page 21: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/21.jpg)
10. Own It
![Page 22: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/22.jpg)
That’s It.
![Page 23: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/23.jpg)
Appendix A. Partial Tool List
Testing: JUnit, Cucumber, NoseBuild / CI: Jenkins, Gradle, JaCoCo
Code Review: GitHubProvisioning: Puppet
Deployment: Fabric, botoMonitoring: Metrics, Graphite
![Page 24: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/24.jpg)
Appendix B. Are You Ready?
Unit Coverage > 90%?
Good Staging Tests?
Informative Monitors?
Builds Are Kept Green?
No API Breaking Changes?
Rigorous Code Reviews?
Support Has Your Phone Number?
Do You Own it?
Not Ready
No Yes
credit: [email protected]
![Page 25: Continuous delivery in practice (public)](https://reader036.fdocuments.in/reader036/viewer/2022062419/558de5031a28abef428b469f/html5/thumbnails/25.jpg)
Thanks. Questions?