Jenkins Automation

Julien Pivotto (@roidelapluie)

Open Source Data Center Conference

Berlin

whoamiJulien "roidelapluie" Pivotto

@roidelapluie

Sysadmin at inuits

Automation, monitoring, HA

Jenkins user for 5+ years

inuits

Jenkins in 2 lines

Open Source

Pluggable

The Jenkins ProjectJava

Hudson (2005)

Jenkins (2011)

The evolution of JenkinsJenkins is moving fast (there is competition:travis-ci, gitlab-runners)

Far away from just the java world (e.g. mvnspecifics are gone in Pipeline world)

What can you do with Jenkins?

Testing, building, deploying

Software

Services

Infrastructure

Mission CriticalPublic Domain https://commons.wikimedia.org/wiki/File:Roaddamage59quake.JPG

Without Jenkins...How do I build this?

Where do I deploy this?

Who knows where to upload this?

When did we deploy this?

Automating JenkinsCreative Commons Attribution-ShareAlike 2.0 https://www.flickr.com/photos/machu/3131057286

Automating Jenkins

Why is it needed?

Make build processes transparent

Make build processes improvable

Make build process reproducible

Lower the barrier to update Jenkins security - audit - bugfixes - plugins

Automating Jenkins

Why is it hard?

Building Software is hard

For long, Jenkins has been strongly UI-Driven

It is "easy" to deploy (.war file) Hey I've deployed Jenkins on my Laptop!

It has lots of plugins (and you need them)

XML everywhere!Creative Commons Attribution 2.0 https://www.flickr.com/photos/generated/6035308729

What can you automate inJenkins?

1. The Operating System Jenkins isrunning on

The platform/node/host/.. which runs yourJenkins damon

2. The Jenkins Service

The start/stop/reload of Jenkins

JAVA_OPTIONS

3. Plugins

Which plugins to install

Which version

Which configuration

4. Jenkins Global Configuration

SMTP Server

Email address

Authentication/Authorization

5. Jobs - Views - Folder

Views - That really help people

Jobs and folders - Creation AND deletion,

6. Jobs definition

Jobs content

Build steps

Deployments configuration

7. Operating system you build yoursoftware on

Jenkins nodes

Where your jobs run

(Linux, Android, ...)

Automating the OSCreative Commons Attribution-Share Alike 2.0 https://www.flickr.com/photos/timdorr/168925884

Jenkins deserves attentionJenkins Server should be automated

Receive OS updates

Be monitored like any other server

Automating the serviceCC0 Public Domain https://pixabay.com/en/butler-tray-beverages-wine-964007/

Installing JenkinsDo NOT use war file (yes I know it's easy)

Use Jenkins LTS (3 months lifecycle vs 1week)

Think about: user, directories, backups,...

Packages provides: upgrade path, downgradepath, control, checksums, signatures, ...

Automating the Jenkins ServiceChef: Jenkins in the supermarket

Puppet: module rtyler/jenkins

Playbooks/etc for other tools as well

Docker

Jenkins PluginsYou NEED them (never seen a Jenkins setupwithout 20-100 plugins)

Fetched from https://updates.jenkins-ci.org

Can be installed from the UI :(

Can be installed from the CLI

Packaging Jenkins PluginsPlugins have dependencies (against plugins &Jenkins core)

They have a fixed download path

They are listed in updates.jenkins.io/update-center.json

https://github.com/roidelapluie/Jenkins-Plugins-RPM

Mirroring Jenkins PluginsMirror http://updates.jenkins-ci.org

By default, "latest" will be fetched

Don't cache too much

github.com/jenkinsci/docker install-plugins.sh

Global configurationCC0 Public Domain https://commons.wikimedia.org/wiki/File:Waiting_room_bell.jpg

Modifying XML files

DON'T

system-config-dsl-pluginLike Job DSL, but for the system

https://github.com/jenkinsci/system-config-dsl-plugin

JENKINS-31094

Downside: it does not exist yet

Creative Commons Attribution-Share Alike 3.0https://commons.wikimedia.org/wiki/File:Groovy-logo.svg

GroovyYet another language to learn? yes.

Programming language for the Java platform

Scripting language

Fully integrated in Jenkins

Used by automation tools (chef cookbook,puppet module...)

The Jenkins Script ConsoleA groovy console is available at /script

http://jenkins.example.com/script

also with curl

Requires "Overall/Run Scripts" permission

Sample Groovy scriptsCreative Commons Attribution-Share Alike 2.0

https://www.flickr.com/photos/bjornb/88101376

Set number of executorsimport jenkins.model.Jenkins;Jenkins.instance.setNumExecutors(0)

Set HTML formatterimport jenkins.model.Jenkins;import hudson.markup.RawHtmlMarkupFormatter;

Jenkins.instance.setMarkupFormatter(new RawHtmlMarkupFormatter(false));

Set system messageimport jenkins.model.Jenkins;Jenkins.instance.setSystemMessage("""Welcome to <em>Jenkins</em>.""");

Configure simple-themeimport jenkins.model.Jenkins;

def simpleThemePlugin = \ Jenkins.instance.getDescriptor( "org.codefirst.SimpleThemeDecorator") simpleThemePlugin.cssUrl = \ "/userContent/example.css"simpleThemePlugin.save()

Everything in $JENKINS_HOME/userContentis served under /userContent

Emailimport jenkins.model.Jenkinsdef desc = Jenkins.instance.getDescriptor( "hudson.tasks.Mailer")desc.setSmtpHost("172.17.0.1")desc.save()

Disable usage statisticsimport hudson.model.UsageStatistics;hudson.model.UsageStatistics.DISABLED = true;

But also...Setup Authorization/Authentication

Setup prefix url

Setup slaves, clouds

Setup global libraries

Setup credentials

Setup first jobs

Bonusimport org.jenkinsci.plugins.pipeline.utility.steps.shaded.org.yaml.snakeyaml.YamlYaml reader = new Yaml()Map config = reader.load(text)

Thanks to pipeline utility step (readYaml()step), you can easily read yaml files

Scripts sourceshttps://wiki.jenkins-ci.org/display/JENKINS/Jenkins+Script+Console

https://github.com/jenkinsci/jenkins-scripts/tree/master/scriptler

https://github.com/infOpen/ansible-role-jenkins/tree/master/files/groovy_scripts

https://github.com/samrocketman/jenkins-bootstrap-jervis/tree/master/scripts

https://github.com/jenkinsci/puppet-

jenkins/blob/master/files/puppet_helper.groovy

http://pghalliday.com/jenkins/groovy/sonar/chef/configuration/management/2014

/09/21/some-useful-jenkins-groovy-scripts.html

Jenkins Javadocshttps://jenkins.io/doc/developer/guides/

http://javadoc.jenkins.io/archive/jenkins-2.32/

http://javadoc.jenkins.io/

http://javadoc.jenkins.io/plugin/gerrit-trigger/

Now what?Creative Commons Attribution 2.0 https://www.flickr.com/photos/51029297@N00/5275403364

Jenkins init.groovy.dUpon startup, Jenkins will run $JENKINS_HOME/init.groovy.d/*.groovy

scripts

Meanwhile, "Jenkins is getting ready..."message is displayed

Drop files, restart Jenkins

Allows you to preconfigure everything --without the GUI

init.groovy.d directorybehaviour

Scripts executed sequentally (alphanumorder)

Scripts that throws exceptions make startupfail

Inside JenkinsCreative Commons Attribution-ShareAlike 4.0 International

https://commons.wikimedia.org/wiki/File:Mystery_Cave_passage.jpg

The Multiple ApproachesGUI .. but this talk is about automation, right?

init.groovy.d: to create your seed job

Jenkins Job Builder: declarative, python, yaml

Jenkins Job DSL: imperative, groovy

Jenkins Job BuilderAn Openstack Project

Python (not a Jenkins Plugin!)

Support templates

Extensible

Can do raw xml

Limited support for plugins and pipeline

Put Jobs config under SCM

Jenkins Job DSLA Jenkins Plugin

2012

Groovy DSL to create views & jobs

Put Jobs config under SCM

init.groovy.ddef jobManagement = new JenkinsJobManagement( System.out, [:], new File('.'))

new DslScriptLoader(jobManagement).runScript("""folder('jenkins') displayName('Jenkins')pipelineJob("jenkins/seed") definition cpsScm scm git remote credentials('jenkins­git­na') url('git.example.com/seed.git') branches('master') scriptPath('Jenkinsfile') """)

Creative Commons Attribution-Share Alike 3.0https://commons.wikimedia.org/wiki/File:Groovy-logo.svg

Job DSL supportLarge community of users

Lots of plugins supported

Create a jobjob('test') scm git('git://example.com/foo.git' 'master') steps shell('make test')

Create a Pipeline jobpipelineJob('test') definition cps script(buildScript)

Set job propertiespipelineJob('test') description('Test Build') parameters booleanParam('verbose', false, 'Be Verbose') logRotator numToKeep(10)

Loopsconfig.each() jobConfig ­> pipelineJob(jobConfig.name) triggers if (jobConfig.triggers?.scm) scm(jobConfig.triggers.scm)

Create a viewlistView('Project A') recurse() jobs regex('myprj/[/]+') columns status() weather() name() lastSuccess() lastFailure() lastDuration() lastBuildConsole() buildButton() progressBar()

Work with pluginsbuildMonitorView("OnScreen Status") jobs name('WatchA')

$JENKINS_URL/plugin/job-dsl/api-viewer/

In PipelinejobDsl removedJobAction: 'DELETE', removedViewAction: 'DELETE', targets: 'seeds/*.groovy'

Unleash the power of groovyimport jenkins.model.Jenkins;

dslCfg = Jenkins.instance.getDescriptor( "javaposse.jobdsl.plugin. GlobalJobDslSecurityConfiguration")dslCfg.useScriptSecurity = false

==

(sec issue with old versions of the plugin)(or power user trick)

Load extra librariesjobDsl additionalClasspath: 'src/yaml.jar', removedJobAction: 'DELETE', removedViewAction: 'DELETE', targets: 'seeds/*.groovy'

Read yamlimport org.yaml.snakeyaml.YamlYaml reader = new Yaml()Map config = reader.load( readFileFromWorkspace('cfg.yaml')

Drop the snakeyaml jar file and add it toclasspath

Jenkins PipelineCreative Commons Attribution 2.0 https://www.flickr.com/photos/amerune/9294639633

Jenkins PipelineJenkins Jobs as Code

"Jenkins file"

Imperative (aka scripted) Pipeline

Declarative Pipeline (2017)

What is a Jenkinsfile (akaPipeline)

A file that contains the definition of a job

No need of Gui

Defines Steps, Reports, Environments,Nodes,...

Plugins can provide steps

Generic "step"

How to write Pipelines?Visual Pipeline editor (WIP)

"Pipeline Syntax" link in jobs

Scriped pipelinenode ("Ubuntu && amd64") stage('checkout') checkout scm stage('build') sh 'make' stage('test') sh 'make test'

Real World Scripted Pipelinenode ("Ubuntu && amd64") checkout scm

stage('build') try sh 'make' catch(err) currentBuild.result = 'UNSTABLE' publishArtifacts('err.log') throw err stage('test') sh 'make test' step([$class: 'JavadocArchiver', javadocDir: "target/site", keepAll: true])

Declarative Pipeline (1/2)pipeline agent label("Ubuntu && amd64") options timeout(time: 235, unit: 'MINUTES') timestamps() ansiColor('xterm') stages stage('build') steps sh('make')

Declarative Pipeline (2/2)pipeline post always junit params.jUnitReports

Declarative vs scriptedDeclarative checkout code by default

Declarative get a workspace "OOTB"

Declarative enforces everything in stages

Declarative allow Pipeline-wide wrappers (e.gansiColor, timestamps)

Going further with PipelineGlobal Libraries Plugins

Job DSL Plugin

Jenkins NodesCC0 Public Domain https://www.flickr.com/photos/133259498@N05/27077266322/

Docker Docker DockerRun jobs inside containers

Clean, short lived containers

Easy to update

Docker Plugin

Docker nodes patternBuild Container

Tag it with a tag "candidate"

Push it to your registry

Run normal testing

Run actual builds with that "candidate"

If success -> tag with "release" && push

In practiceDocker Plugin config automated with groovy

Candidate and Release tags are setup asslaves

They get two labels: "image" "tag" (e.g. "build-centos-7" "candidate")

Jobs get a parameter "tag"

In the build Jenkinsfilepipeline agent label("build­centos­7 && $params.tag")

In the docker-build JenkinsfiledockerImage = docker.build("build­centos­7", "­­no­cache")dockerImage.tag('candidate')docker.withRegistry(url, credentials) dockerImage.push('candidate')

node("build­centos­7 && candidate") sh('test­script')build job: 'myjob', parameters: [string(name:'tag', value:'candidate')] dockerImage.tag('release')docker.withRegistry(url, credentials) dockerImage.push('release')

Pros/ConsUpdated containers won't block the builds (e.gon packages updates)

Containers stay up to date

Don't forget that builds release artifacts(sometime you don't want that in nodes tests)

Docker Moby Docker

Jenkins Master in Docker

WHY WOULD YOU DO THAT?

Atomic upgrade of all plugins at the sametime

Easy rollback / update

Easy to test

Reduce the cost to upgrade

Jenkins master in Docker

At which price?????

Do not run ANYTHING on master; slaveeverything

Keep a Docker Registry with history

Assume Docker instability

Think docker deployment etc... (A pipeline foryour Jenkins master - how cool is this?)

Docker and JenkinsJenkins upstream images: alpine/ubuntu

LTS and weekly

DockerfileFROM jenkins:2.46.2­alpineMAINTAINER dev­team@example.comUSER rootCOPY jenkins/userContent/ \ /var/lib/jenkins/userContentRUN mkdir /var/log/jenkins && \ mkdir /var/lib/jenkins && \ mkdir /var/cache/jenkins chown ­R jenkins: /var/log/jenkins && \ chown ­R jenkins: /var/lib/jenkins && \ chown ­R jenkins: /var/cache/jenkins

Timezone (if needed)

yay alpine

RUN apk add ­­no­cache tzdata && \ cp /usr/share/zoneinfo/Europe/Brussels \ /etc/localtime && \ echo "Europe/Brussels" > /etc/timezone && \ apk del tzdata

Install pluginsUSER jenkinsCOPY jenkins/plugins.txt /tmp/plugins.txtRUN cat /tmp/plugins.txt | xargs \ /usr/local/bin/install­plugins.sh

/usr/local/bin/install-plugins.sh is provided in theupstream docker image

plugins.txt

cloudbees­foldercucumber­reports:4.3.2docker

init.groovy.dCOPY jenkins/init.groovy.d/ \ /usr/share/jenkins/ref/init.groovy.d/

Note: if you keep Jenkins config in a volume, youneed to suffix your files .override to update them

JAVA_OPTSENV JAVA_OPTS="­Xmx8192m \ ­Djenkins.install.runSetupWizard=false"ENV JENKINS_OPTS="­­handlerCountMax=300 \ ­­logfile=/var/log/jenkins/jenkins.log \ ­­webroot=/var/cache/jenkins/war"

Testing the imageJenkins Pipeline

init.groovy.d switch:

if (System.getenv()['DO_NOT_BUILD']) Jenkins.instance.doQuietDown()

Block until startedThat script will fail until Jenkins is started:

#!/bin/bash ­xecurl 127.0.0.1:8080/jenkins2/ |grep 'All \[Jenkins\]' ­q ||grep "WARNING: Failed to run script file" \/var/log/jenkins/jenkins.log ||grep "SEVERE: Failed GroovyInitScript.init" \/var/log/jenkins/jenkins.log

retry(30) sleep 10 sh("""docker exec $dockerContainer.id \ bash ­xe /tmp/block­until­started.sh""")

Testing init.groovy.dset ­xeif grep "SEVERE: Failed GroovyInitScript" \ /var/log/jenkins/jenkins.log || grep "WARNING: Failed to run script file" \ /var/log/jenkins/jenkins.logthen sleep 600 # Let some time to investigate echo "FOUNDÂ ERRORSÂ INÂ THEÂ LOGS" falsefi

Seed jobs

Somewhere in init.groovy.d

cause = new RemoteCause('localhost', 'Build triggered by init.groovy.d')job = Jenkins.instance.getItem("seed")cAction = new CauseAction(cause)build = job.scheduleBuild2(0, cAction)res = build.get().getResult().toString()assert res == 'SUCCESS'

Last lines check for completion of seed job

BenefitsA Pipeline to Deploy Jenkins

Automated checks of init.groovy.d etc

Check that the seed job works

Going further

Will the fun ever stop?

Trash the Jenkins config

Only keep build history

Do notput config in persistent volume

Let init.groovy.d/job dsl build your config onlyfrom code

Still, keep job history

The glueimport jenkins.model.Jenkins;import static groovy.io.FileType.FILES

def rawBuildsBaseDir = '/srv/jenkins/state'

Jenkins.instance.setRawBuildsDir( "$rawBuildsBaseDir/\$ITEM_FULL_NAME")

/srv/jenkins/state is a persistent volume

it will keep build history

ConclusionPublic Domain https://commons.wikimedia.org/wiki/File:YellowOnions.jpg

Jenkins can be FULLYautomated

My recommendations:

init.groovy.d

Jenkins Job DSL

Pipeline

Try dockerized Jenkins ; might work for you(spoiler: works for us)

Julien Pivottoroidelapluie

roidelapluie@inuits.eu

Inuitshttps://inuits.euinfo@inuits.eu

Contact