Monitoring microservices: Docker, Mesos and Kubernetes visibility at scale
Paradigm shift in CI at PayPal with Docker and Mesos
-
Upload
sathiya-narayanan -
Category
Software
-
view
62 -
download
2
Transcript of Paradigm shift in CI at PayPal with Docker and Mesos
©2015 PayPal Inc. Confidential and proprietary.
15th February 2017
Running CI at scale with containers at PayPal
Sathiya Narayanan
© 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 2
At PayPal, we take CI seriously
©2015 PayPal Inc. Confidential and proprietary. 3
o One Ginormous Jenkins Instance, customized for PayPal
o 40,000 runnable jobso 24,000 builds run each day on Single
Instanceo Stack: Java, Node, Python, C++ o CD : Unit Test, Static analysis and code
coverage, & deployments o One Jenkins master with a pool of a
hundred Jenkins slaveso Custom build system for 50 MLOC in <1
Min
Single Jenkins instance
One Giant Jenkins Butler!
©2015 PayPal Inc. Confidential and proprietary. 4
o SPOFo Scalability o Change Managemento No freedom for userso Inefficient resource managemento Plugins don’t scale
Limitations with single Jenkins instance
Too much load!
©2015 PayPal Inc. Confidential and proprietary. 5
o 1:1 modelo Individual VMs running Jenkins for each
Applicationo 2,500+ VMs in useo Each Jenkins instance would have 2
executor on master to perform buildso Freedom for users
Users loved it!
Dedicated VM for each Jenkins Instance
©2015 PayPal Inc. Confidential and proprietary. 6
Jenkins on Mesos
o Light weight o Long running tasko Marathon scheduler
Jenkins Master
o Provisioned on demand with Jenkins Mesos plugin
o Flavors of slaves based on the application stack
Jenkins slave
©2015 PayPal Inc. Confidential and proprietary. 7
Optimized Resource Utilization with Mesos
©2015 PayPal Inc. Confidential and proprietary. 8
Operating cost reduced 10X times
After Mesoso ~2400 CPU’s, o 18 TB, o 7.2 TB RAM
Before Mesoso ~ 24,000 CPU’s, o 180 TB Disk, o 54 TB RAM
©2015 PayPal Inc. Confidential and proprietary. 9
After:
~2400 CPU’s, 18 TB, 7.2 TB RAM
~300K
2.7 million saving could buy:- A personal Jet for GPI
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Recovery was slow without Mesos
©2015 PayPal Inc. Confidential and proprietary. 10
10
After:
~2400 CPU’s, 18 TB, 7.2 TB RAM
~300K
2.7 million saving could buy:- A personal Jet for GPI
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Instant Recovery with Mesos
©2015 PayPal Inc. Confidential and proprietary. 11
After:
~2400 CPU’s, 18 TB, 7.2 TB RAM
~300K
2.7 million saving could buy:- A personal Jet for GPI
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Our workload was too much for Marathon
©2015 PayPal Inc. Confidential and proprietary. 12
o Zero outages and no customer impact.o Aurora was more stableo Aurora was able to scale to more than
5,000 jobs
Replaced Marathon with Aurora
©2015 PayPal Inc. Confidential and proprietary. 13
After:
~2400 CPU’s, 18 TB, 7.2 TB RAM
~300K
2.7 million saving could buy:- A personal Jet for GPI
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million13
CIAPI
Aurora
Mesos Master
Mesos Cluster
Swift / Object Storage
Zookeeper
Jenkins Master Jenkins Slave
Nginx
Highly available setup
Devxdash
©2015 PayPal Inc. Confidential and proprietary. 14
Challengeso Inconsistent build environments between user
workstation and CI build system
o Tasks stepping onto each other
o Providing conflicting tool set for different technology stacks
o Different DevOps teams/applications had their own Mesos cluster
o Slower rollout of new tools to the build environment
o Inconsistent build environments within the CI build system
©2015 PayPal Inc. Confidential and proprietary. 15
Why Docker?
o Task isolation
o Cloud Native
o Eliminates the dependency on the host
o Reproducibility
o Portability
o Immutability
o Homogenous cluster
©2015 PayPal Inc. Confidential and proprietary. 16
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Dockerized Jenkins Masters
o Immutable containers for consistency
o Eliminates the need to have tools installed on slave
o Provides task isolation
o Other work loads can co-exist in Mesos cluster
o Easy to cater to conflicting OS / toolset requirements in a polyglot environment
o Easy to roll out updates
o Other work loads can co-exist in Mesos cluster
o Users can bake their own build environment
Dockerized Jenkins Slaves
Jenkins with Docker
©2015 PayPal Inc. Confidential and proprietary. 17
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Adoption to Docker containers
BC [ Before Containers ]
AD [ After Docker ]
SSH key / Tokens / Maven credentials
Stored in local disk Injected only during build time and rotating them
Toolsets like JDK, Maven etc.
Installed in host Installed within Docker container
©2015 PayPal Inc. Confidential and proprietary. 18
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
What is state for Jenkins master ?
o Jenkins war
o Plugins
o Global configuration
o Job configuration
o Build historyo Console outputo Reports like code coverage, test result trend etc.o Archived artifacts
©2015 PayPal Inc. Confidential and proprietary. 19
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Backing up Jenkins state
Supervisord
Archive the
directory
Encrypt the data
Store in cloud
storage
©2015 PayPal Inc. Confidential and proprietary. 20
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
When Jenkins Master container moves to another VM
Untar the files
into Jenkins home
Decrypt the
backup
Check if a backup exists
in cloud storage
Provisioning script [Entry
point for Docker]
©2015 PayPal Inc. Confidential and proprietary. 21
10X SavingsBefore:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Use case - Maven build
Total build time - 4 minutesPersisting workspace for subsequent builds
Decreased build timeSource code [Clone]
Artifacts
Time taken 0m27.618s
2m56.000sNumber of files 10570 4456
Total Size 158 M 173 M
©2015 PayPal Inc. Confidential and proprietary. 22
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
CIAAS with Docker ArchitecturePublic cloud
Private cloud
©2015 PayPal Inc. Confidential and proprietary. 23
After:
~2400 CPU’s, 18 TB, 7.2 TB RAM
~300K
2.7 million saving could buy:- A personal Jet for GPI
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million