Post on 20-Aug-2015
©2014 CloudBees, Inc. All Rights Reserved
Pimp your Continuous Delivery Pipeline with Jenkins workflow Cyrille Le Clerc, CloudBees
©2014 CloudBees, Inc. All Rights Reserved
About Me
Cyrille Le Clerc
@cyrilleleclerc
Open Source
CTO
DevOps, Infra as Code, Con9nuous Delivery
Product Management
©2014 CloudBees, Inc. All Rights Reserved
About you
Jenkins? CI? CD? Code Jenkins plugin?
©2014 CloudBees, Inc. All Rights Reserved
Agenda
• Con9nuous Delivery • Jenkins & Con9nuous Delivery • Workflows in Jenkins today • The new Jenkins Workflow Engine • Jenkins Workflow Syntax Card • Possible future
©2014 CloudBees, Inc. All Rights Reserved
Continuous Delivery
©2014 CloudBees, Inc. All Rights Reserved
Continuous Delivery
• “Reliable So+ware Releases through Build, Test, and Deployment Automa=on”
• DONE = Shippable into Produc9on
©2014 CloudBees, Inc. All Rights Reserved
Continuous Integration Steps
Con$
nuou
s Integra$o
n Source Code
Quality Analysis
Compile &Unit Tests
Binaries
©2014 CloudBees, Inc. All Rights Reserved
Continuous Delivery Steps
Con$
nuou
s Delivery
DEV
DEV
OPS
QA
Source Code
Quality Analysis
Compile &Unit Tests
Binaries
Perfs Environment
Func Test Environment
UAT Environment
Production Environment
GO LIVE!
Release!
Staging Environment
©2014 CloudBees, Inc. All Rights Reserved
Continuous Delivery Pipeline
Continuous Delivery Pipeline
Compile &Unit Tests
Perfs Tests
Selenium Tests Deploy UAT
Sonar Analysis
Deploy Staging Deploy Production
©2014 CloudBees, Inc. All Rights Reserved
Jenkins: Hub of Continuous Delivery
• Connect the dots
• It is the process Source Code
Perfs Environment
Func Test Environment
UAT Environment
Binaries
Quality Analysis
Compile &Unit Tests
©2014 CloudBees, Inc. All Rights Reserved
CD Pipeline è Workflow
Continuous Delivery Pipeline
Compile &Unit Tests
Perfs Tests
Selenium Tests Deploy UAT
Sonar Analysis
Deploy Staging Deploy Production
©2014 CloudBees, Inc. All Rights Reserved
What we need
• Complex pipelines involving mul9ple stages • Non-‐sequen$al logic such as loops, forks … • Long-‐running builds must survive outages • Interac$ons with humans including pauses, input • Restartable builds in case of transient errors • Reusable defini$ons to avoid duplica9on • Comprehensible scripts with one clear defini9on
©2014 CloudBees, Inc. All Rights Reserved
Workflow until today
• Job chaining
• Build Flow Plugin
• Build Pipeline Plugin
• Build Delivery Pipeline
©2014 CloudBees, Inc. All Rights Reserved
Workflow until today
• Many atomic jobs • Hard to share variables/state between jobs • Limited logic • Mix build triggers, parameterized build …
©2014 CloudBees, Inc. All Rights Reserved
Build Flow Plugin
• Did have scriptability and extensibility • Did not address configura9on “sprawl” • Disjointed view of what really ran • No ability to survive restarts • Almost good enough but could not go further
©2014 CloudBees, Inc. All Rights Reserved
Jenkins Workflow Engine
©2014 CloudBees, Inc. All Rights Reserved
Simplifying our sample workflow
Compile &Unit Tests
Perfs Tests
Selenium Tests Deploy UAT
Sonar Analysis
Deploy Staging Deploy Production
©2014 CloudBees, Inc. All Rights Reserved
Simplifying our sample workflow
Compile &Unit Tests
Perfs Tests
Selenium Tests Deploy UAT
Sonar Analysis
Deploy Staging Deploy Production
©2014 CloudBees, Inc. All Rights Reserved
What we need
• Complex pipelines involving mul9ple stages • Non-‐sequen$al logic such as loops, forks … • Long-‐running builds must survive outages • Interac$ons with humans including pauses, input • Restartable builds in case of transient errors • Reusable defini$ons to avoid duplica9on • Comprehensible scripts with one clear defini9on
©2014 CloudBees, Inc. All Rights Reserved
New Workflow Syntax
©2014 CloudBees, Inc. All Rights Reserved
Key features
• En9re flow is one concise Groovy script – For loops, try-‐finally, fork-‐join …
• Can restart Jenkins while flow is running • Allocate slave nodes and workspaces
– As many as you want, when you want
• Stages throYle concurrency of builds • Human input/approval integrated into flow • Standard project concepts: SCM, ar9facts, plugins
©2014 CloudBees, Inc. All Rights Reserved
Groovy DSL vs. Graphical Workflow
• Familiar control flow construc9on • Familiar “tools” for building abstrac9ons
Func9ons, classes, variables, …
• Workflow in version control • As opposed to:
Graphical workflow designer
©2014 CloudBees, Inc. All Rights Reserved
Project setup
• One workflow is defined as a job • Single script for all steps • Build triggers & parameters like regular projects • Each workflow execu9on is a regular Jenkins build displayed in regular Jenkins views
• Graphical visualiza9on of actual build possible Not of visualiza9on job defini9on but of build execu9on
©2014 CloudBees, Inc. All Rights Reserved
Non-sequential logic
try {} catch {}
©2014 CloudBees, Inc. All Rights Reserved
Interaction with humans
©2014 CloudBees, Inc. All Rights Reserved
Restartable build / checkpoint
©2014 CloudBees, Inc. All Rights Reserved
Visualization
©2014 CloudBees, Inc. All Rights Reserved
Stages
• Semaphore • Visualiza9on
©2014 CloudBees, Inc. All Rights Reserved
Comprehensible scripts
©2014 CloudBees, Inc. All Rights Reserved
Demo time!
©2014 CloudBees, Inc. All Rights Reserved
Workflow Syntax Card
©2014 CloudBees, Inc. All Rights Reserved
Workflow Syntax Card
• stage: Enter a new stage • node: Allocate node • ws: Allocate workspace • parallel: Execute sub-‐workflow in parallel • retry: Retry the body up to N 9mes • catchError: Catch error and con9nue • input: Input / human interac9on • load: Evaluate a Groovy source file into the workflow script
©2014 CloudBees, Inc. All Rights Reserved
Workflow Syntax Card
• step: General build step • sh: Shell script • bat: Windows batch script
©2014 CloudBees, Inc. All Rights Reserved
Workflow Syntax Card
• archive: Archive ar9facts • unarchive: Copy archived ar9fact into the workspace • echo: Print message • dir: Change directory • readFile: Read file from workspace • writeFile: Write file to workspace
©2014 CloudBees, Inc. All Rights Reserved
Workflow Syntax Card
• git: Git • svn: Subversion • scm: General SCM • tool: install a tool • build: Build a job
©2014 CloudBees, Inc. All Rights Reserved
Workflow Syntax Card
• checkpoint: capture the state of the execu9on so that it can be restarted later
©2014 CloudBees, Inc. All Rights Reserved
Possible Future
©2014 CloudBees, Inc. All Rights Reserved
Possible Future: probably OSS
• More plugin interoperability • Mul9-‐branch project (& SCM API) integra9on • More advanced SCM • In-‐IDE edi9ng support • Concise syntax
©2014 CloudBees, Inc. All Rights Reserved
Possible Future: probably Jenkins Enterprise
• More visualiza9ons • Validated merge integra9on • Deployment, incl. blue/green with rollback • Freestyle and/or Build Flow import • Templates integra9on
©2014 CloudBees, Inc. All Rights Reserved
Get the code
• hYps://gist.github.com/cyrille-‐leclerc/796085e19d9cec4a71ef – workflow.groovy
©2014 CloudBees, Inc. All Rights Reserved