CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)
-
Upload
cloudbees -
Category
Technology
-
view
7.871 -
download
1
Transcript of CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)
![Page 1: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/1.jpg)
©2014 CloudBees, Inc. All Rights Reserved
CI and CD Across the Enterprise with Jenkins
Cyrille Le Clerc, CloudBees
![Page 2: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/2.jpg)
©2014 CloudBees, Inc. All Rights Reserved
About Me
Cyrille Le Clerc
@cyrilleleclerc
Open Source
CTO
DevOps, Infra as Code, Con9nuous Delivery
Product Management
![Page 3: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/3.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Agenda
• Con9nuous Delivery & Jenkins Workflow • Scaling Jenkins • Con9nuous Delivery with Jenkins and Puppet and Chef
![Page 4: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/4.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Continuous Delivery
![Page 5: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/5.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Continuous Delivery
• “Reliable So+ware Releases through Build, Test, and Deployment Automa=on”
• DONE = Shippable into Produc9on
![Page 6: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/6.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Continuous Integration Steps
Con$
nuou
s Integra$o
n Source Code
Quality Analysis
Compile &Unit Tests
Binaries
![Page 7: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/7.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Continuous Delivery Steps
Con$
nuou
s Delivery
DEV
DEV
OPS
QA
Source Code
Quality Analysis
Compile &Unit Tests
Binaries
Perfs Environment
Func Test Environment
UAT Environment
Production Environment
GO LIVE!
Release!
Staging Environment
![Page 8: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/8.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Continuous Delivery Pipeline
Continuous Delivery Pipeline
Compile &Unit Tests
Perfs Tests
Selenium Tests Deploy UAT
Sonar Analysis
Deploy Staging Deploy Production
![Page 9: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/9.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Jenkins: Hub of Continuous Delivery
• Connect the dots
• It is the process Source Code
Perfs Environment
Func Test Environment
UAT Environment
Binaries
Quality Analysis
Compile &Unit Tests
![Page 10: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/10.jpg)
©2014 CloudBees, Inc. All Rights Reserved
CD Pipeline è Workflow
Continuous Delivery Pipeline
Compile &Unit Tests
Perfs Tests
Selenium Tests Deploy UAT
Sonar Analysis
Deploy Staging Deploy Production
![Page 11: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/11.jpg)
©2014 CloudBees, Inc. All Rights Reserved
What we need
• Complex pipelines involving mul9ple stages • Non-‐sequen$al logic such as loops, forks … • Long-‐running builds must survive outages • Interac$ons with humans including pauses, input • Restartable builds in case of transient errors • Reusable defini$ons to avoid duplica9on • Comprehensible scripts with one clear defini9on
![Page 12: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/12.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Workflow until today
• Job chaining
• Build Flow Plugin
• Build Pipeline Plugin
• Build Delivery Pipeline
![Page 13: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/13.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Workflow until today
• Many atomic jobs • Hard to share variables/state between jobs • Limited logic • Mix build triggers, parameterized build …
![Page 14: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/14.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Build Flow Plugin
• Did have scriptability and extensibility • Did not address configura9on “sprawl” • Disjointed view of what really ran • No ability to survive restarts • Almost good enough but could not go further
![Page 15: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/15.jpg)
©2014 CloudBees, Inc. All Rights Reserved
New Workflow Syntax
![Page 16: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/16.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Demo time! https://github.com/cyrille-leclerc/spring-petclinic/blob/petclinic-1.0.0-clc/src/main/jenkins/workflow.groovy
![Page 17: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/17.jpg)
©2014 CloudBees, Inc. All Rights Reserved
CD Pipeline è Workflow
Continuous Delivery Pipeline
Compile &Unit Tests
Perfs Tests
Selenium Tests Deploy UAT
Sonar Analysis
Deploy Staging Deploy Production
![Page 18: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/18.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Key features
• En9re flow is one concise Groovy script – For loops, try-‐finally, fork-‐join …
• Can restart Jenkins while flow is running • Allocate slave nodes and workspaces
– As many as you want, when you want
• Stages throYle concurrency of builds • Human input/approval integrated into flow • Standard project concepts: SCM, ar9facts, plugins
![Page 19: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/19.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Groovy DSL vs. Graphical Workflow
• Familiar control flow construc9on • Familiar “tools” for building abstrac9ons
Func9ons, classes, variables, …
• Workflow in version control • As opposed to:
Graphical workflow designer
![Page 20: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/20.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Project setup
• One workflow is defined as a job • Single script for all steps • Build triggers & parameters like regular projects • Each workflow execu9on is a regular Jenkins build displayed in regular Jenkins views
• Graphical visualiza9on of actual build possible Not of visualiza9on job defini9on but of build execu9on
![Page 21: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/21.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Non-sequential logic
try {} catch {}
![Page 22: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/22.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Interaction with humans
![Page 23: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/23.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Restartable build / checkpoint
![Page 24: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/24.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Visualization
![Page 25: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/25.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Stages
• Semaphore • Visualiza9on
![Page 26: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/26.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Comprehensible scripts
![Page 27: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/27.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Scaling with Jenkins
![Page 28: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/28.jpg)
©2014 CloudBees, Inc. All Rights Reserved
• How do I scale Jenkins from 10 to 1000 jobs? Projects? Users?
• How do I comply with corporate
rules?
• How do I keep Jenkins flexible for project teams?
Problem Statement: Scale!
![Page 29: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/29.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Typical Jenkins installations
Let’s think about this for a few minutes
![Page 30: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/30.jpg)
©2014 CloudBees, Inc. All Rights Reserved
The top down “structured” and “lets grow vertically” approach
5 projects 0 Slaves
25 projects 5 Slaves
75 projects 50 Slaves
150 projects 400 Slaves
![Page 31: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/31.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Vertical Scaling issues
When is the next failure? Single point of failure
![Page 32: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/32.jpg)
©2014 CloudBees, Inc. All Rights Reserved
The “organic” agile growth – “let each group do what they want to” approach
Horizontally… virally
![Page 33: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/33.jpg)
©2014 CloudBees, Inc. All Rights Reserved
• Share executors between masters – Not possible today
Horizontal Issues: I Would Like to…
Master-‐b
Master-‐a
Master-‐c
![Page 34: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/34.jpg)
©2014 CloudBees, Inc. All Rights Reserved
• Share security information – Security realm and roles
• Push configurations & upgrades
Master-‐b
Master-‐a
Master-‐c
Horizontal Issues: I Would Like to…
![Page 35: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/35.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Jenkins Opera9ons Center
oc.cloudbees.com Sample Topology
Master-‐b
Master-‐a
Master-‐c
Shared Roles , Security Realm and versions
AYached-‐master client
AYached-‐master client
AYached-‐master client
pushed to client master
pushed to client master
![Page 36: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/36.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Demo time!
![Page 37: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/37.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Continuous Delivery traceability with Jenkins and puppet and Chef
xxx Environment
![Page 38: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/38.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Puppet environments
Prod style environment: – Master & agents
Dev Environment
Puppet Agentpetclinic.war
Application Serverwar file + puppet manifests
& modules
puppet apply
“Production style” Environment
Puppet Master
Puppet Agentpetclinic.war
Application Serverwar file + puppet manifests
& modules
Dev environment: – Standalone agents + “puppet apply”
![Page 39: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/39.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Test environment with Puppet and Vagrant
Dev Environment
Puppet Agentpetclinic.war
Application Serverwar file + puppet manifests
& modules
vagrant provisionpuppet apply
Vagrant to create “clean” puppet managed servers
![Page 40: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/40.jpg)
©2014 CloudBees, Inc. All Rights Reserved
CI with Jenkins and Vagrant and Puppet
Jenkins builds war file and puppet modules and deploys via vagrant
puppet deployment
modulejava code Dev Environment
Puppet Agentpetclinic.war
Application Serverwar file + puppet manifests
& modules
vagrant provisionpuppet apply
![Page 41: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/41.jpg)
©2014 CloudBees, Inc. All Rights Reserved
CI with Jenkins and Vagrant and Puppet
Jenkins builds war file and puppet modules and deploys via vagrant
puppet deployment
modulejava code Dev Environment
Puppet Agentpetclinic.war
Application Serverwar file + puppet manifests
& modules
vagrant provisionpuppet apply
puppet deployment
modulejava code Dev Environment
Puppet Agentpetclinic.war
Application Serverwar file + puppet manifests
& modules
vagrant provisionpuppet apply
notification
![Page 42: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/42.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Jenkins – Puppet Integration
![Page 43: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/43.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Demo time! https://github.com/CloudBees-community/vagrant-puppet-petclinic/tree/vagrant-puppet-petclinic-0.1.0 jenkins-workflow-with-vagrant-and-puppet.groovy: https://gist.github.com/cyrille-leclerc/7f621c341e193622c844
![Page 44: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/44.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Jenkins & Puppet
• Jenkins Deployment No9fica9on Plugin – hYps://wiki.jenkins-‐ci.org/display/JENKINS/Deployment+No9fica9on+Plugin
• Puppet module for Jenkins tracking – hYps://github.com/jenkinsci/puppet-‐jenkinstracking
• Puppet Labs plugin for Jenkins – hYps://wiki.jenkins-‐ci.org/display/JENKINS/Puppet+Plugin
• CloudBees DevOps page – hYps://developer.cloudbees.com/bin/view/Main/DevOps
• This demo – hYps://github.com/CloudBees-‐community/vagrant-‐puppet-‐petclinic – Jenkins Workflow with Vagrant and Puppet: hYps://gist.github.com/cyrille-‐leclerc/7f621c341e193622c844
![Page 45: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/45.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Jenkins & Chef?
• Similar concepts • Jenkins Deployment No9fica9on Plugin
– hYps://wiki.jenkins-‐ci.org/display/JENKINS/Deployment+No9fica9on+Plugin
• Puppet module for Jenkins tracking – hYps://github.com/jenkinsci/puppet-‐jenkinstracking
• Jenkins Chef Tracking – hYps://wiki.jenkins-‐ci.org/display/JENKINS/Chef+Tracking+Plugin
• CloudBees DevOps page – hYps://developer.cloudbees.com/bin/view/Main/DevOps
![Page 46: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/46.jpg)
©2014 CloudBees, Inc. All Rights Reserved
Conclusion
• Jenkins workflow to build Con9nuous Delivery pipelines
• Jenkins Opera9ons Center by CloudBees to scale Jenkins
• Puppet & Chef deployments can be integrated in Jenkins
![Page 47: CI and CD Across the Enterprise with Jenkins (devops.com Nov 2014)](https://reader033.fdocuments.in/reader033/viewer/2022060205/55a09def1a28ab6b6a8b4861/html5/thumbnails/47.jpg)
©2014 CloudBees, Inc. All Rights Reserved