An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines

An Open-SourceChef Cookbook CI/CD ImplementationUsing Jenkins Pipelines

Steffen Gebert (@StGebert)

Config Management Camp, Gent, 06.02.2017

2

Agenda

• Context

• Motivation

• Jenkins Pipelines (in general)

• Chef CI/CD using Jenkins Pipelines

About MeResearcher / PhD Student(Software-based Networks)2011 - 2016

Core Team Member2010 - 2013

Server Admin Team Membersince 2011

3

Co-Organizer DevOps Meetup Würzburgsince 2016

4

PHP-based Content Management System

Picture by Benjamin Kott

5

TYPO3 Open Source Project

• Established open-source community since 1997

• TYPO3 Association as non-profit organization

• Server Admin Team responsible for project infrastructure *

Thanks for partiallyfunding my work andcovering travel costs!

*listedonopensourceinfra.org

Picture by Daniel Pötzinger: http://www.typo3-media.com/blog/article/typo3-wallpaper-for-wide-screen.htmlBased on image by Mike Swanson: http://interfacelift.com/wallpaper/details.php?id=1402 Pi

ctur

e by

Ste

fan

Buse

man

n

6

Chef at TYPO3

• Started with Chef in 2011• Started with librarian-chef (monolithic repo)• Took years to get rid of monolithic repo

• Currently operating 45 Debian nodes• Usual tooling: berkshelf, test-kitchen• berkshelf-api-based /universe (connecting to Chef Server 11*)

• Small team of volunteers, varying degree of Chef knowledge

* because of laziness

7

Demo Time!

Picture by annca/ pixabay:https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/

Working with Chef(and knife, and berks, and..)

9

We haveworkflow problems!

Picture by tpsdave / pixabay:https://pixabay.com/en/firefighters-fire-flames-outside-115800/

12https://blog.twitter.com/2017/the-infrastructure-behind-twitter-scale

13

Related Work

• zts/cooking-with-jenkins• Provides resource to create single Jenkins freestyle jobs per cookbook• Executes pre-defined rake tasks

• chef-solutions/pipeline• Reads cookbooks from a Berksfile enclosed in main chef-repo• Creates single Jenkins freestyle job per cookbook

• Chef Automate• Commercial• Little known to me, what's going on under the hood

14

Survey

• Who’s Chef cookbooks (Puppet modules/etc) are tested in CI?

• Who has an automated release process?

15

Jenkins Pipelines

16

Jenkins Pipelines

Jenkins PipelinesIntroduction to

20

Jenkins Pipeline DSL

• Groovy DSL• Provides steps, like sh• Defined in the Jenkinsfile• Can use Groovy/Java logic (Chef users like that idea, right?)

• Pipeline plugin (workflow-aggregator)• Formerly called "Workflow"• Open-sourced one year ago• Whole bunch of plugins (10+)

sh "make"sh "make install"

21

Jenkins Jobs as Code?

• Jenkins Job Builder• Python / YAML based• From OpenStack

• Job DSL plugin (job-dsl)• Also Groovy DSL• Supports many, many plugins• Creates only single jobs, not pipelines

# Job DSL examplejob('my-project-main') {scm {git('https://github.com/...')

}triggers {scm('H/15 * * * *')

}publishers {downstream('my-project-unit')

}}

22

Cookbook Pipeline?

sh 'berks install'sh 'foodcritic'sh 'kitchen test'sh 'berks upload'

Picture by BarnImages / pixabay:https://pixabay.com/en/sushi-food-japanese-fish-seafood-789820/

23

Stages

• Allow visual groupingstage('lint') {sh 'foodcritic'sh 'cookstyle'

}stage('resolve dependencies') {sh 'berks install'

}stage('test-kitchen') {sh 'kitchen test'

}stage('upload') {sh 'berks upload'

}

24

Nodes

• Allocates a Jenkins executor (master/slave)

• Optionally with label

node {stage('lint') {sh '..'

}stage('resolve') {sh '..'

}}

node('chefdk') {stage('lint') {..}stage('resolve') {..}

}

Master Agent2Agent1

Executors(node):

Pipeline

Stage

Step

Step

Stage

Step

Step

Stage

Step

Step

25

Ready, Set, Go!

26

Multibranch Jobs

• Scans repo for branches containingJenkinsfile• Automatically creates (deletes) jobs

• Works with plain Git• Works better™ with GitHub / Bitbucket (via API calls)

27

More Steps to Come..

• Jenkins Plugins can contribute DSL steps

28

Global Variables

• Environment variables:

• Information about current build:

• Parameterized build*:

env.BRANCH_NAMEenv.BUILD_URL

params.myparamparams.deployEnv

* well-hidden featuresee https://st-g.de/2016/12/parametrized-jenkins-pipelines

currentBuild.displayName = "v1.2.3"CurrentBuild.rawBuild.getCause()currentBuild.result = 'FAILED'

29

Now Copy & Paste?

Picture by aitroff / pixabay:https://pixabay.com/en/stormtrooper-star-wars-lego-storm-1343772/

30

Pipeline Shared Libraries

• Additional code to be used by Jenkinsfiles• Loaded from SCM repo

• Inclusion via• @Library annotation in Jenkinsfile• Configuration of enclosing folder• Jenkins global configuration*

* then library code is "trusted"

31

Configuring Global Pipeline Library

• Jenkinsfile:

• Load implicitly would load it without @Library• Use non-default branch/tag

@Library('chefci')org.example.Pipeline ...

@Library('chefci@testing')

Don't playwell together

33

Global Variables

• Can store global state (for the current build)• Can behave like steps• Useful to simplify pipeline code• Hide implementation details from users

# vars/deployTo.groovydef call(def environment) {echo "Starting deployment to ${environment}"withCredentials(..) {sh "rsync –avzh . ${environment}.example.com"

}}

# Jenkinsfilenode {sh "make test"deployTo "production"

}

34

More Magic: Global Library Classes (src/)

• Groovy classes implementing arbitrary logic • Make use of pipeline DSL steps• Use other Jenkins functions• Import and use Java libraries

à How we implement our pipeline

35

# Declarative Jenkinsfilepipeline {agent label:'has-docker', dockerfile: trueenvironment {GIT_COMMITTER_NAME = "jenkins"

}stages {stage("Build") {steps { sh 'mvn clean install' }

}stage("Archive"){// ..

}}post {always {deleteDir()

}success {mail to:"[email protected]", subject:"SUCCESS// ..

Scripted vs. Declarative Pipelines

• Scripted pipelines• Just (imperative) Groovy code• The original implementation• The approach used here

• Declarative pipelines• Hit the 1.0 release last Friday• Can be validated prior to execution• Ease some tasks, i.e., failure handling• Visual Pipeline Editor plugin

36

Jenkins Pipeline Summary

• Groovy DSL for specifying pipelines as code

• Ideally stored within the tested repo

• Extract shared functionality to library repos

• Pipeline execution triggered on push event

jenkins-chefciAn Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines

38

Forbidden Commands$ berks upload$ knife cookbook upload$ knife data bag from file$ knife data bag delete$ knife environment from file$ knife environment delete$ knife role from file$ knife role delete

Allowed Commands

$ git pull$ git commit$ git push

Pictures by PublicDomainPictures & serrano / pixabay:https://pixabay.com/en/costume-demon-devil-board-female-15747/https://pixabay.com/en/bebe-girl-child-child-portrait-1237704/

39

Meet the jenkins-chefci cookbook

• Sets up a Chef CI/CD infrastructure using Jenkins• Sets up Jenkins master• Installs ChefDK, configures credentials• Configures job that scans a GitHub organization• Configures our shared pipeline library

https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/

• Add this Jenkinsfile to your cookbooks:# Jenkinsfile@Library('chefci') _org.typo3.chefci.v2.cookbook.CookbookPipeline.builder(this, steps)

.buildDefaultPipeline().execute()

40

Meet the Shared Pipeline Library

• Implements pipelines for• Cookbook testing, versioning & upload• [WIP] Chef-repo's data bags, environments, roles

• Contains couple of Groovy classes• Some helper classes• *Pipeline as entry points from Jenkinsfile• AbstractStage as base class for.. Stages

• Yes, we're at v2 already J

41

Executed Stages & Steps

• Linting• Foodcritic• Cookstyle

• Build• Berkshelf install (endpoints for Supermarket & Chef Server)Berksfile.lock in Git for top-level cookbooks

• Acceptance• Test-Kitchen (using kitchen-docker)

• Publish• Version Bump (thor-scmversion, based on user feedback)• Berkshelf upload (to Chef Server)

Picture by Kaz/ pixabay:https://pixabay.com/en/hands-hand-raised-hands-raised-220163/

43

node {createKitchenYaml() stash "cookbook"

}

parallel("essentials-debian-86": { node {

unstash "cookbook"sh "kitchen test --destroy always essentials-debian-86" }},

"essentials-ubuntu-1604": { node { .. } },"full-debian-86": { node { .. } },"full-ubuntu-1604": { node { .. } }

)

Test-Kitchen

• If there is no .kitchen.docker.yml, put default one• Use Jenkins' parallel step

expected result, not hard-coded

44

Test-Kitchen (2)

• Parallel instances automatically derived from kitchen status

• Log files of failed instances are archived• Less chaos than complete Jenkins log• Traditional UI does not clearly show failed parallel branch

• Not limited to kitchen-docker

45* cookbook configures this via API

49

The Art of Cookbook Versioning

• Let's agree on SemVer• I do not agree with manual changes to metadata.rb

50

Demo Time!

Picture by annca/ pixabay:https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/

51

Versioning

• Based on thor-scmversion• Made by RiotGames for their cookbooks• Uses Git tags to derive current version• Version treated as "label", not part of the source code

• Jenkins pushes only Git tags, no code changes

# metadata.rbname "example-cookbook"version IO.read(File.join(File.dirname(__FILE__), 'VERSION')) rescue '0.0.1'

52

Versioning (2)

• Publish stage notifies via Slack

• Except commit message subject indicates version increase• Do this #patch• Add this #minor• Break this #major

53

"Main Chef-Repo"

• Not covered here, but in site-chefcitypo3org cookbook• Freestyle job, defined via JobDSL• Uses script to parse git diff and

upload changes using knife

54

Additional Notes

• Manually configured J organization-level webhook triggersimmediate builds• Everything else is done automatically, no "manual handholding"

Using jenkins-chefci

56

Ease of Use

• Checkout cookbook from github.com/TYPO3-cookbooks/jenkins-chefci

• Increase GitHub API rate limit

• Allow to steal your Chef credentials

• Run test-kitchen

export JENKINS_GITHUB_LOGIN=johndoeexport JENKINS_GITHUB_TOKEN=123456supersecure

export JENKINS_COPY_CHEF_CREDENTIALS=1

kitchen converge full-ubuntu-1604

WARN: allows commitstatus update

WARN: allows knife/berksto access Chef Server

57

(Potential) TODOs for you

• Write your wrapper cookbook around jenkins-chefci• Point it to your organization• Add authentication, e.g., GitHub OAuth?

• Fork the global library• Adjust pipeline to your needs• You don't have to agree with our (current) implementation

58

Outlook / TODOs

• Documentation / blog posts• Move more stuff from site-chefcitypo3org to jenkins-chefci• chefdk() global function to run in chef/chefdk Docker container• Store Chef private key as Jenkins credential• Test using multiple chef-client version• Use JobDSL for organization-folder setup (#966)• Trigger downstream cookbooks

• Read out dependencies and subscribe to successful pipeline runs• Paves the road to Policyfiles

• Use Jenkins slaves• Collect chef-client deprecation warnings

59

Conclusion

• Jenkins Pipelines allow to define pipelines as code• Groovy-based DSL allows programming of pipelines• Can definitively become complex, i.e., debugging

• Jenkins Pipelines for Chef cookbook CI/CD• Running at TYPO3 since May 2016 (site-chefcitypo3org cookbook)• Public instance at https://chef-ci.typo3.org, open source from day 1• Warning: Many cookbooks still use v1 pipeline (git-flow)

• jenkins-chefci cookbook as reusable implementation• Makes setup accessible to broader audience• No dependencies to other TYPO3 cookbooks

60

Further Reading• TYPO3's Chef CI:

https://chef-ci.typo3.org• TYPO3-cookbooks on GitHub:

https://github.com/TYPO3-cookbooks

• TYPO3's Shared Global Library:https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/

• Pipeline Tutorial:https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md

• Getting started with pipelines:https://jenkins.io/pipeline/getting-started-pipelines/

• Step documentation:https://jenkins.io/doc/pipeline/steps/

• Pipeline shared libraries:https://jenkins.io/doc/book/pipeline/shared-libraries/

• Notifications (Mail, Slack, etc.) in Scripted Pipelines:https://jenkins.io/blog/2016/07/18/pipline-notifications/

• Declarative Pipelineshttps://jenkins.io/blog/2016/12/19/declarative-pipeline-beta/https://jenkins.io/blog/2017/02/03/declarative-pipeline-ga/

An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines

Technology

Transcript of An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines