Breaking the Monolith: Organizing Your Team to Embrace Microservices
-
Upload
paul-osman -
Category
Software
-
view
297 -
download
0
Transcript of Breaking the Monolith: Organizing Your Team to Embrace Microservices
“Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.”
— Melvyn Conway, 1967
500px Team
• Approx 40 full-time employees• Approx 20 full-time engineers + 6 designers / ux researchers• Product Teams
- iOS & Android (3)- Prime (2)- 500px Web (4)- Platform Engineering (Operations + Dev) (7)- QA / Release Engineering (3)
January 2014
• Classic web architecture (LB, App Servers, DB Servers).
• Many data stores, one large monolithic codebase.
• Scaling: all or nothing.
• Cascading failures. Lots of SPOFs.
• Large, unwieldy codebase.
• “Get in line” approach to deployments.
Search & Indexing
• Indexing is naturally asynchronous.
• Introduced dependency on RabbitMQ.
• Wrote new indexer in Go. First service and first Go project.
- Huge performance improvement. 20 hours to 20 minutes.
• Search service - HTTP service that translates query strings into ES queries.
• Eliminated major SPOF. ElasticSearch / Search outages no longer affect customers.
• Isolated knowledge of data store (ElasticSearch).
Uploads & Resizing
• Perfect for first service to move to EC2.
• Continued in Go, because it was working well for us.
• Performance improvement. Upload times dropped significantly.
- Bulk uploads via Lightroom plugin.
• Eliminated another SPOF. Next step: front-end work to gracefully communicate to user.
• Eliminated write-half of S3 knowledge. Read-half still to be done.
Activities & Notifications
• Ongoing.
• Replacing poorly designed service built on top of MongoDB.
• Will eliminate another SPOF. Fix a lot of reliability complaints.
• Get to revisit architecture, centre around data needs (both real-time and ETL workflows).
Where to Start
• Pick a business capability that is:
• Off the Critical Path
• Can be double-written to easily (i.e. indexing, aggregating, etc)
• Look out for natural integration points between disparate systems — also good candidates
Design for Business Capabilities
• Break your application into business capabilities (i.e. search, recommendations, uploads).
• Tempting to design around technology layers (front-end, back-end). Don’t.
• Even if you don’t (yet) have the numbers, treat each business capability as a team.
Team Structure
• Web App Team - Own front-end code, web-bff API facade
• Search Team - Own search infrastructure & service
• Media Team - Own upload service, resizing & converting services, watermarking, etc
• Mobile Team - Own mobile apps, mobile-bff API facade
• Platform Eng - Everything else, operations, devtools, QA / Release engineering