www.camunda.comtwitter.com/camundaBPM

18.09.15

Christian LipphardtSebastian Menski

Berlin Expert Days 2015

From Virtual Machines to Containers

www.camunda.com

Agenda

• Introduction• The Dark Age• The Promising Present• Lessons Learned• The Bright Future

-2-

www.camunda.com

Introduction - About Camunda

-3-

2014201320122011201020092008

IncorporationCamunda Services GmbHBerlin, Germany

IncorporationCamunda Inc.San Francisco, US

BPM Consulting BPM Software Vendor

● 30 Fulltime Employees ● No external funding ● Avg. 50% Turnover Growth per Year

Introduced Hudson

Adapted Jenkins

www.camunda.com

Introduction - Camunda BPM

-4-

www.camunda.com

Introduction - What is Camunda BPM

• Camunda BPM is an open source platformfor workflow and business process automation

• Integrates with:• 7 Application Server (11 different versions)• 6 Databases (17 different versions)• 1 Development & 4 Maintained Versions

-5-

www.camunda.com

Introduction - Why CI is Important

• Every Camunda BPM version is tested against:• 187 combinations of DBs and App Servers• 11 JDKs• ~ 400 Jobs per version

• Bi-Annual release of a new Camunda BPM version• Support for Enterprise Customer (24/7 + Fix Time)

-6-

www.camunda.com

Why our CI had to changeThe Dark Age

-7-

www.camunda.com

The Dark Age - The Numbers

• 1 Jenkins Master with lots of plugins• 8 Jenkins Slaves VMs• ~1000 Jobs total configured /

manually managed:• 4 Camunda Versions• Community Projects• Websites• Maintenance

-8-

www.camunda.com

The Dark Age - Isolation Problem

• Unit and Integration Tests need a database/application server

• Only 1 Instance per Database• All Jobs use the same Databases• Every half-year a new Version (~400 Jobs) using same

Databases

-9-

www.camunda.com

The Dark Age - No Scalability

• Total Executors: 12• 1 Jenkins -> 4 Executors• 8 static heterogenous slave VMs, each with 1 Executor

• Jobs tied to slaves through labels• Slaves “restrict” database access by allowing no other build

to run

-10-

www.camunda.com

The Dark Age - Maintenance Problems

• Upgrading Jenkins or any plugin• Supporting a new Database vendor / version• Supporting a new App Server version• Creating jobs for new Camunda BPM version• Disaster recovery

-11-

www.camunda.com

The Dark Age - The Other Problems

• Slow feedback cycle for developers• Developers cannot reproduce CI environments• QA engineers use and maintain separate bloated test build

setup

-12-

www.camunda.com

So …

-13-

www.camunda.com

How we solved our ProblemThe Promising Present

-14-

www.camunda.com

The Present - What we achieved

• Configuration & Infrastructure as Code• Isolated and Reproducible Jobs• Scalable CI Infrastructure

-15-

www.camunda.com

The Present - Infrastructure as Code

1. Every configuration is checked into SCM2. Every application/test runs in a Docker Container3. Every Docker image is build automatically

-16-

www.camunda.com

The Present - Infrastructure as Code

1. Every Configuration is checked into SCM

• Docker for• Applications• Test Environments

• JobDSL for• Jenkins Jobs

-17-

www.camunda.com

The Present - Example - JobDSL

package ci.community

import generator.JobGeneratorimport util.*

Server.COMMUNITY().each { server -> new JobGenerator( name: "${Version.CURRENT}-webapp-IT-${server}", pollScmSchedule: null, pollRemoteJob: RemoteJobHelper.WEBAPP_DISTRO, rootPom: 'qa/pom.xml', goals: "clean verify -P${server},h2,webapps-integration", envVariables: ['DISPLAY': ':0'], jdkVersion: Tool.JDK_FOR_SERVER(server), label: Label.CHROME ).build(this)}

-18-

www.camunda.com

The Present - Infrastructure as Code

2. Every application/test runs in a Docker Container

Images:• Application (Jenkins, Nexus …)• Test Env. Images (DB + SSH)• Build Env. Images

• DIND, QEMU + Packer.io

-19-

www.camunda.com

The Present - Example - DB2

FROM ci1.camunda.loc:5000/camunda-ci-base-centos

ENV DB2_VERSION=10.5 DB_USERNAME=camunda DB_PASSWORD=camunda DB_NAME=engine DB2_HOME=/opt/ibm/db2/V10.5 DB_DIR=/db2_databasesRUN save-env.sh DB2_VERSION DB_USERNAME DB_PASSWORD DB_NAME DB2_HOME

RUN install-packages.sh libaioADD etc/db2/db2expc.rsp /tmp/db2expc.rsp

RUN curl https://nginx.service.consul/ci/binaries/ibm/db2/v10.5_linuxx64_expc.tar.gz > /tmp/db2.tar.gz && \ mkdir -p /tmp/db2 && tar xzf /tmp/db2.tar.gz -C /tmp/db2 --strip 1 && \ cd /tmp/db2 && ./db2setup -r /tmp/db2expc.rsp && rm -rf /tmp/db2.tar.gz /tmp/db2 && \ mkdir ${DB_DIR} && chown db2inst1:db2iadm1 ${DB_DIR} && chmod 775 /${DB_DIR} && \ su -l db2inst1 -c "mkdir ${DB_DIR}/engine"

ADD bin/* /usr/local/bin/ADD etc/supervisor.d/db2.conf.ini /etc/supervisord.d/

EXPOSE 50000

-20-

www.camunda.com

The Present - Infrastructure as Code

3. Every Docker Container is build automatically

• Own Jenkins for Docker/KVM Images• KVM Images build in Docker Container

with Packer + QEMU• KVM Images bundled in Docker Image

-21-

www.camunda.com

The Present - Example - SQL Server 2012

FROM ci1.camunda.loc:5000/ubuntu:14.04.2

ENV IMAGE_NAME=sqlserver-2012.qcow2 IMAGE_DIR=/qemu/ RUN_DIR=/qemu/run

RUN mkdir -p $IMAGE_DIR $RUN_DIRVOLUME $RUN_DIRADD bin/* /usr/local/bin/

RUN apt-get update && \ apt-get -y install --no-install-recommends curl qemu-system-x86 qemu-utils && \ apt-get clean && rm -rf /var/cache/* /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN curl https://nginx.service.consul/ci/binaries/microsoft/${IMAGE_NAME} > ${IMAGE_DIR}/${IMAGE_NAME}

EXPOSE 22 1433 5900

CMD ["/usr/local/bin/start-qemu.sh"]

-22-

www.camunda.com

The Present - The Current Flow

-23-

camunda-ci

camunda

Camunda BPMPlatform

InfrastructureJenkins

CI Jenkins

www.camunda.com

The Present - Isolation

One Jenkins per Concern:

• CI• Release• Infrastructure• Community and other Projects• Marketing

-24-

www.camunda.com

The Present - Isolation & Reproducibility

• Every Jobs runs in an One-Shot Docker Container• No Interference between Jobs• The Database Settings are well documented• Every Docker Image is stored in a private registry• Developers/QA can use the Docker Images for local testing

-25-

www.camunda.com

The Present - Scalability

• Jenkins uses Docker-Plugin with one Docker Cloud running on Docker Swarm

• Docker images are added through Groovy scripting• Running on Commodity Hardware

• 3 Infrastructure Hosts (Jenkins, Nexus, …)• 4 Docker Hosts as 1 Swarm

-26-

www.camunda.com

The Present - Example - Jenkins Docker Cloud

images.each { imageName, imageValues -> imageValues.versions.each { version -> def templateBase = createLinuxTemplateBase(image, privileged) def image = "${registryUrl}/${imageName}:${version}".toString() def label = getLabel(imageValues.label, version) def privileged = imageValues.privileged? imageValues.privileged : false def remoteFs = '/home/camunda' def instanceCap = linuxInstanceCap

def template = new DockerTemplate(templateBase, label, remoteFs, remoteFs, instanceCap, Node.Mode.EXCLUSIVE, 1, createDockerSshLauncher(), createRetentionStrategy(), true, DockerImagePullStrategy.PULL_ALWAYS) dockerTemplates << template }}

-27-

/var/lib/jenkins/init.groovy.d/dockercloud.groovy

www.camunda.com

The Present - Scalability

-28-

www.camunda.com

The Present - CI Cluster

-29-

Docker

Consul

Swarm

Nexus

Registry

ci1

Docker

Docker

Docker

Consul

Swarm

4x Jenkins

ci2

Docker

Docker

Consul

Swarm

Nginx

ci3

Docker

ci4-7

BIND

Infrastructure Build Slaves

C-Agent

S-Agent

www.camunda.com-30-

Jenkins Demo

www.camunda.com

The Present - Advantages

• Easy to add new Databases/Test Environments• New Release = New Branch of JobDSL Repository• Fully parallelized Job Execution• Accountable Configuration History• Testable Infrastructure• Minimize Administration Overhead

-31-

www.camunda.com

The Present - Conclusion

• 2 People + 3 Months of Work• A fully scalable, isolated and reproducible CI Infrastructure• Faster Feedback• Happy Developers and Product Owner

-32-

www.camunda.com

Some advice from us to youLessons learned

-33-

www.camunda.com

Lessons learned - Architecture

• Automate as much as you can• Jenkins config• Jobs config• Environment creation

• Design to scale to support the business agility

-34-

www.camunda.com

Lessons learned - Testing

Test everything:• Jenkins configuration

• required plugin features• Job generation• Docker images• Scalability• Disaster recovery

-35-

www.camunda.com

Lessons learned - Job DSL

• Unit-test the job generation• Write JobGenerator classes to abstract the common build

logic of most jobs out of the box• Use XML diffing to compare previously generated jobs with

new ones

-36-

www.camunda.com

Lessons learned - Docker

-37-

https://github.com/rancherio/vm/

www.camunda.com

Lessons learned - Plugins

• Pin your plugin versions• Be prepared to contribute to plugin development or

maintain a branch yourself• Choose the right plugin for the job

Our Top 3 plugins:JobDSL, Docker-Plugin, Build-Failure-Analyzer

-38-

www.camunda.com

Lessons learned - Control

• Control as much as possible• Third party binaries vs package manager• explicit versions• own mirrors for important packages

-39-

www.camunda.com

What comes nextThe Bright Future

-40-

www.camunda.com

The Future

-41-

• Public Community Jenkins• Internal Webapp for Developers and QA to start

Environments (Dattln)• Continuous Deployment for Infrastructure Container• Back to the Datacenter• Centralized Logging and Monitoring (ELK)• Secret management

www.camunda.com-42-

Logstash Plugin

www.camunda.com-43-

www.camunda.com-44-

www.camunda.com

Resources

• Camunda CI repositoryhttps://github.com/camunda-ci

• Job DSL gradle examplehttps://github.com/sheehan/job-dsl-gradle-example

• Camunda BPM Docker imagehttps://registry.hub.docker.com/u/camunda/camunda-bpm-platform/

-45-

www.camunda.com

We Are Hiring

Come Join Us@camundaBPM

http://camunda.com/jobs

-46-