Scaling Jenkins with Kubernetes 1.2

About meSenior DevOps Engineer @ Glide

rubygems.org/profiles/amimahloof

Contribution to OpenSource via RubyGems and GitHub:

github.com/innovia

linkedin.com/in/amimahloof

Glide environment

Multiple Redis ServersMySQLDynamoDBMultiple Background Queue ProcessorsApp ServerMultiple Web Socket ServersCloudSearch

Previous Jenkins setupSingle Jenkins Master machine for Server code buildsSingle Jenkins Master Machine for Android client builds

Dedicated MySQL database per build

Dedicated port per build serviceEach build environment encapsulated and managed by Eye - process monitoringhttps://github.com/kostya/eye

Previous Jenkins IssuesUnder-utilized instances running 24/7 (expensive!)Port collisionsOut of memory issues - build failuresDatabases needed to be created and wiped per buildDebugging failed tests was extremely hardUpdating Jenkins ec2 image especially for Android was a challenge

NFS-SERVERPOD EBS Volume

Scaling Jenkins with Kubernetes

Jenkins Master BackEnd RC

Jenkins Master Android RC

Backend POD

Jenkins Ingress Controller

POD

nginx configmap

Jenkins SlavePOD

With KubeCtl

Backend Service (IP)

single elb for both CI’s

addresses

Current Jenkins Kubernetes Plugin

https://github.com/jenkinsci/kubernetes-pluginSupport single docker image(jenkins-slave docker image - it create’s a POD with that image)

Does not support multiple containers in a POD

Does not support Persistent Volumes

Modified Jenkins Kubernetes Pluginhttps://github.com/innovia/kubernetes-plugin

Support reading and parsing a POD template fileSupport multiple containers in a PODSupport Persistant Volumes, EBS, EmptyDir, HostPath VolumesDoes not support POD template per JOB (System global config)

Read POD Template

Start Build

Send POD Object

Jenkins SlavePOD

download JNLP file from Jenkins Master

Connect to Jenkins Master Slave Port

Build Flow Part 1

Pull Request

Build Flow Part 2

Jenkins Master Jenkins Slave

Start Job on Slave

git fetch and merge server code

create serviceJOB_NAME-BUILD_NUMBER

create PV - PVCJOB_NAME-BUILD_NUMBER start tests

teardown POD, SERVICE, PV, PVC

clean-up temp location

submit result to bitbucket

create PODJOB_NAME-BUILD_NUMBER wait for setup complete file

(check dependancies)

AdvantagesOn Demand PODs Complete test isolationSource controlled by developers flexible POD templateJobs queued up in Kubernetes Scheduler until node resources are available (running builds finish)Scalable Kubernetes Nodes Plugin keeps track of POD - if POD dies in the middle it launches up a new slave for that job Persistant Storage (EBS via NFS)Spot instances using node selectors

Kubernetes on Spot Instances

Save 70-90% over On Demand

Managed by Spot Fleet to reduce downtimeDifferent fleets for different types of Kubernetes Nodes, allowing dynamic pod allocation using Node Selector

Until implemented in Kubernetes 1.3, we have to create our own Fleets

eg.

https://github.com/kubernetes/kubernetes/issues/24472#issu

ecomment-211975112

Thank you!

Questions?Slides available

http://www.slideshare.net/AmiMahloof/scaling-jenkins-with-kubernetes