Netflix Open Source: Building a Distributed and Automated Open Source Program
-
Upload
aspyker -
Category
Technology
-
view
1.902 -
download
0
Transcript of Netflix Open Source: Building a Distributed and Automated Open Source Program
![Page 1: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/1.jpg)
Netflix Open Source
Andrew Spyker (@aspyker) - Engineering Manager
Building a distributed andautomated open source program
![Page 2: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/2.jpg)
About Netflix
● 86.7M members● A few thousand employees● 190+ countries● > ⅓ NA internet download traffic● 500+ Microservices● Many 10’s of thousands VM’s● 3 regions across the world
![Page 3: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/3.jpg)
![Page 4: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/4.jpg)
Trivia
Netflix been opensourcing, since?
a) Around the start of streaming service - 2007b) Around when we went international - 2010c) Around House of Cards release time - 2013
![Page 5: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/5.jpg)
Answer
2010
![Page 6: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/6.jpg)
Why does Netflix Open Source?
Improve Engineering● Great feedback from wider community● Collaborate through open code
Recruit new and retain engineering talent● Hard problems are openly worked on
![Page 7: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/7.jpg)
Industry Alignment
Why does Netflix Open Source?
Netflix movesto cloud
2008
2013
2016
![Page 8: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/8.jpg)
http://netflix.github.io
![Page 9: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/9.jpg)
Open Source Functional Areas
● Contribute to Hadoop, Hive, Pig, Parquet, Presto, Spark● Genie - RESTful API’s for Big Data Jobs● Lipstick - Graphical depiction of executing Pig jobs● Aegisthus - Data pipeline from Cassandra to Big Data
![Page 10: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/10.jpg)
Open Source Functional Areas
● Nebula - Plugins for gradle to simplify builds● Animator - Bakes AMI’s from OS installation packages● Spinnaker - New continuous delivery platform
![Page 11: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/11.jpg)
Open Source Functional Areas
● Eureka, Ribbon, Hystrix - Cloud native, resilient IPC● Karyon, Prana, Archius - Microservice App Frameworks● Fenzo - Mesos advanced scheduling library
![Page 12: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/12.jpg)
Open Source Functional Areas
● Photon - Java Interoperable File Format implementation● VMAF - Perceptual quality metric algorithm and test toolkit
![Page 13: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/13.jpg)
Open Source Functional Areas
● Raigad/Priam - Management/ops sidecars for ES and C*● EVCache - Distributed, replicated memcache++● Dynomite - Dynamo layer on top of non-dynamo data stores
![Page 14: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/14.jpg)
Open Source Functional Areas
● Spectator/Atlas - Monitoring and Telemetry client and server● Vector - Fine grained per instance performance monitoring● Vizceral - Worldwide traffic to microservice graph
visualization● Simian Army - Suite of automations and resiliency testing
![Page 15: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/15.jpg)
Open Source Functional Areas
● Security Monkey - Automated cloud security monitoring● Scumblr/Sketchy - Internet intelligence gathering● FIDO - Security event orchestration (analysis/response)● Lemur - Simplified x.509 cert management● Sleepy Puppy - Delayed cross site scripting framework
![Page 16: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/16.jpg)
Open Source Functional Areas
● Work across front end technologies including Restify● Falcor - Virtual JSON graph & optimized query to backends● RxJS - Simplify Javascript async event based programming
![Page 17: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/17.jpg)
Netflix’s approach to open source
Form a small cross-functional team working group that centralizes OSS competence, assisting decentralized teams working with OSS spend less time focusing on the administrative aspects (legal, tooling, branding, monitoring, and community promotion).
![Page 18: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/18.jpg)
Open source enabler - OSS Interest Group
● Internal mailing list● Meets once per month● Topics from developers● Help each other with
common problems
![Page 19: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/19.jpg)
Trivia
How many OSS projects does Netflix have?
a) 59b) 102c) 176
![Page 20: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/20.jpg)
Answer
176
Netflix (119) Spinnaker (17) nebula-plugins (40)
![Page 21: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/21.jpg)
Open Source Shepherds
● Management with business context
● Consistency across related projects
● Document how area fits together
● Focus on OSS health of each area
![Page 22: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/22.jpg)
Common tools accelerate developers
● Security● Backup● Github user/group repo management● Project tracking● Build systems● CI systems
![Page 23: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/23.jpg)
Security tools
● We scan code for○ Access keys, credentials, email
addresses, hostnames● Provide tools and automation to
○ Scan before initial release○ Scan repeatedly on github
![Page 24: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/24.jpg)
Source code management
● Backup and archival○ Github down != Netflix down
● Internal mirrors we could build from
![Page 25: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/25.jpg)
Project Ownership
All projects have● Development lead, Management lead● Shepherd from OSS function areaOnly projects with active leads stay active!
![Page 26: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/26.jpg)
Github management
● Has to be easy○ Otherwise, teams will go it alone
● Has to be automated○ Self service - chat ops○ Following secure best practices
![Page 27: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/27.jpg)
Github user managementSupport bring github id● User links to internal id● All tools then can
associate identity
Two Factor Auth Enforcement● Automation to boot users who don’t● Be careful - education on recovery!
![Page 28: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/28.jpg)
Github group management
● Owners○ Limited group - due to power○ Automate via chatops all owner actions
● Netflixer group○ Full write permissions on all repos
● Outside contributors○ Added by netflixers, validated over time
![Page 29: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/29.jpg)
Github automated through chat ops
![Page 30: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/30.jpg)
Overall Org Health Tracking
Metrics we track
● Issues○ open, closed, TTC
● Pull Requests○ open, closed, TTC
● Last commit timing● Stars/forks● Num contributors
![Page 31: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/31.jpg)
Project Health Tracking
github.com/Netflix/
OSSTracker
![Page 32: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/32.jpg)
● Repeatable builds● deb/rpm files for OS
package baking● Reduces boilerplate for
common best practices● Standards for
release/version mgmt
Common Build For Gradle/Java
nebula-plugins.github.io
![Page 33: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/33.jpg)
Common CI Systems
● Travis CI○ Populate .travis.yml and sh files○ Standard targets for snapshots,
candidates, and releases○ Binary upload credentials handled○ Consistency across projects
● Cloudbees○ Job-dsl to create release jobs
![Page 34: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/34.jpg)
Using Docker to make projects easier
● A running image is worth a thousand wiki documents
● Started with ZeroToDocker○ Monolithic solution○ Leveraged Dockerhub
trusted builds
![Page 35: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/35.jpg)
Introducing TravisCI Docker buildsFunction Dockerhub
trusted buildsTravisCI Docker support
Github commit traceable builds ✔ ✔
Trusted build servers ✔ ✔
Full build control (labels, etc.) ✖ ✔
Easy to integrate with artifact releases ✖ ✔
● Experimenting: OSSTracker & Genie● Docker compose used across images
![Page 36: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/36.jpg)
TODO Group
● Joined 2015● Collaborate on how
to better collaborate● Leverage TODO group’s work
○ Github focus○ Automation innovations
● Good group for helping OSS companies
![Page 37: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/37.jpg)
Trivia
Which of the followingdoes Hystrix lead in?
a) Most PR’s closed d) Most Forksb) Most Issues closed e) Most contributorsc) Most Stars
![Page 38: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/38.jpg)
Answer
All of the above
![Page 39: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/39.jpg)
Recent NetflixOSS Releases
CI atNetflix scale
Multi-region deployment control
Advanced CI/CD pipelines
![Page 40: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/40.jpg)
Recent NetflixOSS Releases
Chaos Monkey 2.0● Integrated with Spinnaker● Termination scheduling better● Termination event tracking
Photon● Java IMF implementation● Parsing, Interpretation, Validation
![Page 41: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/41.jpg)
Recent NetflixOSS Releases
Vizceral● React and Web Component● Graph data to visualize traffic
Dynomite● Dynamo layer on top of data stores● Redis and memcache● Manager (config, multi-region, backup)
![Page 42: Netflix Open Source: Building a Distributed and Automated Open Source Program](https://reader034.fdocuments.in/reader034/viewer/2022052514/5870d90a1a28ab64768b7399/html5/thumbnails/42.jpg)
Questions?
Andrew Spyker (@aspyker) - Engineering Manager