Continuous Deployment Applied at MyHeritage
-
Upload
ran-levy -
Category
Engineering
-
view
464 -
download
0
Transcript of Continuous Deployment Applied at MyHeritage
![Page 1: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/1.jpg)
Continuous Deployment AppliedRan Levy, Backend DirectorElad Shmitanka, Operations engineer
![Page 2: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/2.jpg)
Agenda
● Overview about MyHeritage
● Background – the days before CD
● Why switching to CD?
● CD
● Wins
![Page 3: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/3.jpg)
Family history for Families
Building next generation tools for family history enthusiasts and their families
Discover Preserve Share
![Page 4: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/4.jpg)
Challenge: Scale
79 million registered users
1.9 billion tree profiles
6.2 billion historical records
200 million photos
42 languages
1 million daily emails
![Page 5: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/5.jpg)
Agenda
● Overview about MyHeritage
● Background – the days before CD
● Why switching to CD?
● CD
● Wins
![Page 6: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/6.jpg)
Background – the days before CD
● Working in branches (many).
● Weekly service pack (dedicated branch).
● Emergencies and HOT Service Pack.
![Page 7: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/7.jpg)
Background – the days before CD
● Advantages:○ Intensively tested and monitored.
● Disadvantages:○ Delivering value to user only on weekly basis. ○ Unstable deliveries to QA without clear owner to problems.○ Developers needs to get back to previous work.○ Huge time waster across the entire R&D.○ Difficult rollbacks in case a problem reached production.
![Page 8: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/8.jpg)
Agenda
● Overview about MyHeritage
● Background – the days before CD
● Why switching to CD?
● CD
● Wins
![Page 9: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/9.jpg)
What is Continuous Deployment ?
Continuous Deployment is a set of practices aimed at, building, testing, and releasing software frequently.
These principles help reduce the cost, time and risk of delivering changes to customers by allowing for more incremental changes to applications in production.
![Page 10: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/10.jpg)
Why switching to CD?
● Fast feedback loop.
● Risk reduction.
● Better coding.
● Increase velocity.
● Easy and fast recovery.
● Bridges the gap between QA (team) and Dev.
![Page 11: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/11.jpg)
Agenda
● Overview about MyHeritage
● Background – the days before CD
● Why switching to CD?
● CD ○ Transition phase
○ The early days
○ The future is here
● Wins
![Page 12: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/12.jpg)
The transition phase
Before switching to CD
● Learn from others (like we did).● Several engineering practices and tools MUST be in
place.
![Page 13: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/13.jpg)
The transition phase
![Page 14: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/14.jpg)
The transition phase
![Page 15: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/15.jpg)
The transition phase
![Page 16: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/16.jpg)
The transition phase
![Page 17: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/17.jpg)
The transition phase
● Gradually skipping Service Pack○ No actual gain for SPCs (manual dists).○ We gave up SPCs and the sky didn’t fall.○ Still coding in branches.
● Small gradual steps:○ Applying CD in completely new code by a single dev.○ Applying CD in a single agile team.○ Applying CD in two agile teams.
![Page 18: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/18.jpg)
The transition phase
● What have we learned?○ Fewer bugs.○ More stability in production.○ Better velocity.
![Page 19: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/19.jpg)
CD – the early days
● More frequent commits. ● Branches have gradually disappeared.● Manual procedure for updating production
○ Prone to human errors○ Required dist synchronization○ Time waster○ …
● Let’s improve and automate the process
![Page 20: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/20.jpg)
CD – the future is here
![Page 21: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/21.jpg)
What did we have?
● Servers list - Static list● Scripts - Mixture of PHP and bash● Error handling - Manual ● SVN problems - Calculating deltas, long processes, conflicts● Dist method - Rsync , only delta of files● Queue
![Page 22: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/22.jpg)
● Scripts - Jenkins with a few scripts
Ok, So what did we change?
● Servers list - Mcollective using Puppet filters
● Error handling - Jenkins Flow plugin, catch● SVN problems - Working on trunk, revert & update● Dist method - RPM, Mcollective● Queue - Builtin in Jenkins
![Page 23: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/23.jpg)
What did we add?
● Tests● Apache configuration changes● Notifications - In Hipchat, with mentioning● Daily digest of changes● Automatic cleanup of the build machine
![Page 24: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/24.jpg)
So, how does it looks like? (Hipchat)
![Page 25: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/25.jpg)
And in jenkins?
![Page 26: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/26.jpg)
Flow schema
![Page 27: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/27.jpg)
Flow schema
![Page 28: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/28.jpg)
Flow schema
Prepareworkspace
![Page 29: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/29.jpg)
Flow schema
Prepareworkspace
Run Tests
Prepare assets
![Page 30: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/30.jpg)
Flow schema
Run Tests
Prepare assets
Suit 1
Suit 2
Suit n
Build RPM
IntegrationCanary
![Page 31: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/31.jpg)
Flow schema
Run Tests
Suit 1
Suit 2
Suit n
Integration
Dist
![Page 32: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/32.jpg)
Flow schema
Suit 1
Suit 2
Suit n
Integration
Dist
CleanupHandle flow results
![Page 33: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/33.jpg)
Flow schema
Prepare workspace
Parse commit message
Run Tests
Build RPM
Canary Integration
Handleflow results
Dist Cleanup
Suit 1Suit 2Suit n
Prepare assets
![Page 34: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/34.jpg)
Drilldown
● Jenkins & Groovy hacks● RPM● MCollective● Hipchat integration● Emergency job
![Page 35: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/35.jpg)
Jenkins & Groovy hacks
● Accessing all the classes of jenkins● How do we make sure the SVN revision will be static across all the jobs?
![Page 36: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/36.jpg)
Jenkins & Groovy hacks
● Accessing all the classes of jenkins● How do we make sure the SVN revision will be static across all the jobs?● How do we know which files changed?
Flow #9 Flow #8 Flow #7 Flow #6
Prepare workspace
Prepare workspace
Prepare workspace
Prepare workspace
Flow #5
Prepare workspace
![Page 37: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/37.jpg)
RPM
RPM (RedHat Package Manager) - Package management system for RedHat (Originally). Contains arbitrary set of files, configurations files and pre & post scripts.
![Page 38: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/38.jpg)
RPM (continue)
● Why RPM? (In short? a lot)○ Mature○ Config files are managed/tracked○ Version tracking○ Dependency management○ Native OS tools to manage lifecycle (install/query/update/uninstall/downgrade)○ Rich ecosystem and toolchain○ Always contains the entire codebase (easier to recover from missed updates)○ Doesn’t touch unmanaged files (i.e PID files)
● Problems we have encountered..○ Large packages (Reduced from a ~700M to currently ~450M)○ I/O & Network usage on the repo machine (simple HTTP server)○ Yum locking mechanism in Puppet
![Page 39: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/39.jpg)
MCollective
MCollective - a framework for building server orchestration or parallel job-execution systems. Most users programmatically execute administrative tasks on clusters of servers.
![Page 40: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/40.jpg)
MCollective (Continue)
● Packages plugin - https://github.com/myheritage/mcollective-plugin-packages
● Distributor plugin - In-house○ Used for emergency dists (explained later)○ clear cache/reload apache
● Dynamic host list○ Easier to manage - Given free by Mcollective○ Host in maintenance - Simply stop Mcollective service
● Scaleable
![Page 41: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/41.jpg)
HipChat
Group and private chat, file sharing, and integrations.
● Has API● Web, Mobile & desktop clients● Mentioning● History● Rooms
![Page 42: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/42.jpg)
HipChat (Continue)
● Using HipChat plugin V0.1.8● Plugin allows only limited functionality (0.1.9 offers more), No
customized messages, no mentioning● Groovy for the rescue!● HuBot for the rescue!
![Page 43: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/43.jpg)
Emergency job
We have problems in the site, what do we do?
1. Put a stop flag - Disabling new dists2. Committing a fix and disting emergency
![Page 44: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/44.jpg)
Emergency job
Get changed files Compress Upload to httpd
“Go, download and extract”
![Page 45: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/45.jpg)
Additional problems we’ve encountered
● Parallelism of UnitTests● Minify failures● Stop flag job● Clear cache
○ PHP is script based language○ Cache is used to improve performance○ requires cache invalidation
![Page 46: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/46.jpg)
CD 2.0 / Lessons learned
● Improving visibility of the root cause● Break the Groovy to files and methods● Yum locking (Should be resolved at Puppet 4.x)● RPM has it’s disadvantages
○ MCollective RSync plugin (https://github.com/myheritage/mcollective-rsync-agent)
![Page 47: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/47.jpg)
Agenda
● Overview about MyHeritage
● Background – the days before CD
● Why switching to CD?
● CD
● Wins
![Page 48: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/48.jpg)
Wins
● Around 20-30 dists per day to deliver close feedback and higher business value.
● Reduced maintenance time for dist procedure.● Higher quality:
○ Less bugs.○ Better coding.○ Increased testing coverage.
![Page 49: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/49.jpg)
Wins
● Reduced code base and assets separation from code base.
● Higher velocity.
● Easy and fast recovery.
● Satisfaction or R&D, DevOps and the organization.
![Page 50: Continuous Deployment Applied at MyHeritage](https://reader034.fdocuments.in/reader034/viewer/2022051122/58e833481a28ab1c5c8b5ab9/html5/thumbnails/50.jpg)
We are hiring!