Building A Distributed Build System at Google Scale (StrangeLoop 2016)
-
Upload
aysylu-greenberg -
Category
Software
-
view
505 -
download
1
Transcript of Building A Distributed Build System at Google Scale (StrangeLoop 2016)
![Page 1: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/1.jpg)
Building a Distributed Build System
at ScaleAysylu Greenberg
September 17, 2016
![Page 2: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/2.jpg)
Aysylu Greenberg
@aysylu22
![Page 3: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/3.jpg)
Building Distributed Build System
at Google Scale
![Page 4: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/4.jpg)
Building Distributed Build System
at Google Scale
![Page 5: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/5.jpg)
What does a build system do?Photo by smemon / CC BY
![Page 6: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/6.jpg)
Build Scenario
Project with dependencies
Makefilemain.o : main.c defs.h cc -c main.cmain.c : curl –O http://remote/main.c
![Page 7: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/7.jpg)
Build Scenario
Project with dependencies
Find dependenciesMakefilemain.o : main.c defs.h cc -c main.cmain.c: curl –O http://remote/main.c
![Page 8: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/8.jpg)
Build Scenario
Project with dependencies
Find dependencies
Build project with the dependencies
Download build artifacts
![Page 9: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/9.jpg)
Test Scenario
Project with dependencies
Find dependencies
Build project with the dependencies
Download build artifacts Run the test
Get the results of the test
![Page 10: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/10.jpg)
Building Distributed Build System
at Google Scale
![Page 11: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/11.jpg)
Building Distributed Build System
at Google Scale
![Page 12: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/12.jpg)
Scale• Engineers: >30,000 developers in 40+ offices
![Page 13: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/13.jpg)
Scale• Engineers: >30,000 developers in 40+ offices• Commits: 15K by humans + 30K by robots/day
![Page 14: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/14.jpg)
Scale• Engineers: >30,000 developers in 40+ offices• Commits: 15K by humans + 30K by robots/day• Source code: 2 billion LOC
![Page 15: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/15.jpg)
Scale• Engineers: >30,000 developers in 40+ offices• Commits: 15K by humans + 30K by robots/day• Source code: 2 billion LOC• Builds and tests: 5M per day through BuildRabbit
![Page 16: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/16.jpg)
Scale• Engineers: >30,000 developers in 40+ offices• Commits: 15K by humans + 30K by robots/day• Source code: 2 billion LOC• Builds and tests: 5M per day through BuildRabbit• Petabytes of output artifacts
![Page 17: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/17.jpg)
Scale• Engineers: >30,000 developers in 40+ offices• Commits: 15K by humans + 30K by robots/day• Source code: 2 billion LOC• Builds and tests: 5M per day through BuildRabbit• Petabytes of output artifacts• 1 repository
![Page 18: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/18.jpg)
Working in One RepositoryPhoto by flyingblogspot / CC BY SA
![Page 19: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/19.jpg)
Repository of Artifacts vs Building from SourcePhoto by flyingblogspot / CC BY SA
![Page 20: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/20.jpg)
Working in One Repository • Linear revision history
![Page 21: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/21.jpg)
Working in One Repository • Linear revision history• Extensive code sharing & reuse
• Everything can be cross-referenced• Large-scale refactoring for codebase modernization• Easier collaboration across teams
![Page 22: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/22.jpg)
Working in One Repository • Linear revision history• Extensive code sharing & reuse• Simplified dependency management
• No confusion about authoritative version of the file• No forking of shared libraries• Components for library releases = Git subtree or Git
subcomponents to separate release from WIP versions
![Page 23: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/23.jpg)
Working in One Repository • Linear revision history• Extensive code sharing & reuse• Simplified dependency management• Predictable, repeatable builds from source
![Page 24: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/24.jpg)
Working in One Repository • Linear revision history• Extensive code sharing & reuse• Simplified dependency management• Predictable, repeatable builds from source• Optimizations to avoid compiling same artifacts
![Page 25: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/25.jpg)
Working in One Repository • Linear revision history• Extensive code sharing & reuse• Simplified dependency management• Predictable, repeatable builds from source• Optimizations to avoid compiling same artifacts• Decouple each team's processes as much as
possible
![Page 26: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/26.jpg)
Building Distributed Build System
at Google Scale
![Page 27: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/27.jpg)
Building Distributed Build System
at Google Scale
![Page 28: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/28.jpg)
Build System StoryPhoto by imanka / CC BY
![Page 29: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/29.jpg)
Towards Distributed Build System
![Page 30: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/30.jpg)
Towards Distributed Build System
![Page 31: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/31.jpg)
Towards Distributed Build System
![Page 32: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/32.jpg)
Towards Distributed Build System
![Page 33: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/33.jpg)
Towards Distributed Build System
![Page 34: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/34.jpg)
Towards Distributed Build System
![Page 35: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/35.jpg)
BuildRabbit:Distributed Build System
![Page 36: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/36.jpg)
BuildRabbit: Distributed Build System
IN THE CLOUD
![Page 37: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/37.jpg)
What does BuildRabbit do?
![Page 38: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/38.jpg)
Continuous integration
system
BuildRabbit
Source System
Google engineers &
teams
Release infrastructure
Integration testing
infrastructure
Build artifact storage
Blaze
![Page 39: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/39.jpg)
Building Distributed Build System
at Google Scale
![Page 40: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/40.jpg)
Building Distributed Build System
at Google Scale
![Page 41: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/41.jpg)
From Push to Pull: Client / Server
Client forUser
BuildRabbitScheduler
BuildRabbitWorker
![Page 42: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/42.jpg)
From Push to Pull: Build Service
User
Persistent Queue
BuildRabbit Worker
Build ArtifactsBuild Progress
Info
eventRPCstream
![Page 43: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/43.jpg)
From Push to Pull: Build Service
User
Persistent Queue
BuildRabbit Worker
Build ArtifactsBuild Progress
Info
eventRPCstream
![Page 44: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/44.jpg)
Client for
User
BuildRabbitScheduler
BuildRabbitWorker
![Page 45: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/45.jpg)
Client for
User
BuildRabbitScheduler
BuildRabbitWorker
User
Persistent Queue
BuildRabbit Worker
Build Progress InfoBuild Artifacts
![Page 46: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/46.jpg)
Client for
User
BuildRabbitScheduler
BuildRabbitWorker
User
Persistent Queue
BuildRabbit Worker
Build Progress InfoBuild Artifacts
RESILIENCE LOGIC
RESILIENCE LOGIC
![Page 47: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/47.jpg)
Continuous integration
system
BuildRabbit
Source System
Google engineers &
teams
Release infrastructure
Integration testing
infrastructure
Build artifact storage
Blaze
![Page 48: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/48.jpg)
Migration with Zero DowntimePhoto by moonjazz / CC BY SA
![Page 49: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/49.jpg)
MIGRATE BACKENDS FIRST
![Page 50: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/50.jpg)
Client for
User
BuildRabbitScheduler
BuildRabbitWorker
User
Persistent Queue
BuildRabbit Worker
Build Progress InfoBuild Artifacts
![Page 51: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/51.jpg)
FOCUS ON MIXED MODEMIGRATE BACKENDS FIRST
![Page 52: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/52.jpg)
TARGET LAUNCH-FRIENDLY CLIENTSFOCUS ON MIXED MODEMIGRATE BACKENDS FIRST
![Page 53: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/53.jpg)
Client for
User
BuildRabbitScheduler
BuildRabbitWorker
User
Persistent Queue
BuildRabbit Worker
Build Progress InfoBuild Artifacts
![Page 54: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/54.jpg)
DECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTSFOCUS ON MIXED MODEMIGRATE BACKENDS FIRST
![Page 55: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/55.jpg)
Client for
User
BuildRabbitScheduler
BuildRabbitWorker
User
Persistent Queue
BuildRabbit Worker
Build Progress InfoBuild Artifacts
![Page 56: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/56.jpg)
MAXIMUM VISIBILITY INTO SYSTEM STATEDECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTSFOCUS ON MIXED MODEMIGRATE BACKENDS FIRST
![Page 57: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/57.jpg)
PRACTICE THE LAUNCHMAXIMUM VISIBILITY INTO SYSTEM STATEDECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTSFOCUS ON MIXED MODEMIGRATE BACKENDS FIRST
![Page 58: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/58.jpg)
PRACTICE THE LAUNCH, A LOTMAXIMUM VISIBILITY INTO SYSTEM STATEDECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTSFOCUS ON MIXED MODEMIGRATE BACKENDS FIRST
![Page 59: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/59.jpg)
HAVE A SOLID ROLLBACK PLANPRACTICE THE LAUNCH, A LOTMAXIMUM VISIBILITY INTO SYSTEM STATEDECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTSFOCUS ON MIXED MODE
![Page 60: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/60.jpg)
Distributed System Design PatternsPhoto by oslointhesummertime / CC BY
![Page 61: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/61.jpg)
Distributed vs Centralized Control PlanePhoto by oslointhesummertime / CC BY
![Page 62: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/62.jpg)
Distributed Design Patterns: Distributed Control
Design implementation is scaled from 1 machine
Bigger overhead during evolution
![Page 63: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/63.jpg)
Distributed Design Patterns: Centralized Control
Distributed design from the start
Bigger overhead during design phase
![Page 64: Building A Distributed Build System at Google Scale (StrangeLoop 2016)](https://reader031.fdocuments.in/reader031/viewer/2022012923/587fac531a28ab107e8b4985/html5/thumbnails/64.jpg)
Building a Distributed Build System
at ScaleAysylu Greenberg
@aysylu22