Post on 15-Apr-2017
Jenkins Pipeline @ ScaleBuilding Automation Frameworks for Systems Integration
CloudBees | Jenkins Automotive and Embedded DayBöblingen, April 05, 2017
Oleg Nenashev, CloudBees
© 2017 CloudBees, Inc. All Rights Reserved. 2
About me. Hardware/Embedded
@oleg_nenashev
oleg-nenashev LibreCoresSt. Petersburg Polytechnic University
© 2017 CloudBees, Inc. All Rights Reserved. 3
About me. Jenkins project
@oleg_nenashev
oleg-nenashev
© 2017 CloudBees, Inc. All Rights Reserved. 4
Oleg’s “Hall of Shame”(c)
• Jenkins Core• Remoting• Plugins
© 2017 CloudBees, Inc. All Rights Reserved. 5
2016
http://bit.ly/nenashev-automotive-day-2016
© 2017 CloudBees, Inc. All Rights Reserved. 6
ØHow/why do we build frameworks for Embedded?ØWhat has changed in Pipeline since 2016?ØHow to effectively use and develop Pipeline?ØLessons learnt
Agenda
© 2017 CloudBees, Inc. All Rights Reserved. 7
• Pipeline 101• Pipeline Generic Features
• Blue Ocean• Declarative Pipeline
• Automation Infrastructure• Agents, Tools, Hardware Peripherals
• Added-value Pipeline features in CloudBees Jenkins Enterprise
What is NOT in the talk?
Ask the Experts,Jenkins Online Meetup
Birds of Feather,Ask the Experts
© 2017 CloudBees, Inc. All Rights Reserved. 8
What is a typical Embedded Product?
`Hardware
External IPFirmware / Embedded
SW
External SW
Development tools / EDAs
Demo software Demo hardware
Developer kits
“Main product”
Documentation
© 2017 CloudBees, Inc. All Rights Reserved. 9
Embedded area. Welcome to the automation hell!
• Almost everything needs to be automated
• Multiple technologies and tools
• Integration testing is important
• Unstable hardware & networks
• Legacy stuff is everywhere
• …
© 2017 CloudBees, Inc. All Rights Reserved. 10
OK, Automotive…
Source: http://www.popsautoelectric.com/auto-electric/
• System of systems• High level of integration• Components by 3rd
parties• Many standards• High quality requirements
© 2017 CloudBees, Inc. All Rights Reserved. 11
1. The most popular CI automation server in the world
2. It’s a framework => flexible• Swiss Army Knife of Automation
3. It’s open source => customizable4. Commercial support is available5. Useful plugins
Why Jenkins?
© 2017 CloudBees, Inc. All Rights Reserved. 12
Система
Docs
Plugins Configs
Infra-structure
SamplesDemos
How do we buildCI Systems?
Infra-structure
Infra-structure
• Agents (FKA ”slaves”)• Tools• Hardware peripherals• Computing grids
© 2017 CloudBees, Inc. All Rights Reserved. 13
Approach: Using “classic” Job types:• Freestyle, Matrix, JobDSL, …
Classic Approaches
Keywords:• Script steps to run the logic• Custom Tools Plugin to manage tools• Throttle Concurrent Builds and Lockable
Resources to dispatch access to hardware• Naginator Plugin to restart builds on-demand• Parameterized Trigger / Copy artifacts to
parallelize tests and to collect results
JUC2015/London
https://jenkins.io/solutions/embedded/
© 2017 CloudBees, Inc. All Rights Reserved. 14
Classic Approaches
Spaghetti Automation
• Barely traceable dependencies
• Complex build flow• Complex
configurations • Duplication
• Expensive maintenance
© 2017 CloudBees, Inc. All Rights Reserved. 15
ü Store build definitions with the projectü Track changes together with the projectü Tests together with the project
Configuration as Code
© 2017 CloudBees, Inc. All Rights Reserved. 16
Configuration as Code in Jenkins
Job ConfigurationJenkins system configuration
© 2017 CloudBees, Inc. All Rights Reserved. 17
Jenkins System Configuration… as Code
External Tools
Jenkins CLI and REST
API
python-jenkins
jenkins-client (java)
Configuration Management
systems
Ansible, Chef
Docker
Solutions within Jenkins
project
Groovy Boot Hooks
Scriptler Plugin
SCM Sync Configuration
UnstableJust examples…
© 2017 CloudBees, Inc. All Rights Reserved. 18
Configuration as Code for jobs
Job DSL
Job Builder Plugin
Jenkins PipelinePipeline is offered by default in Jenkins 2.0
…
Groovy
YAML
Groovy
© 2017 CloudBees, Inc. All Rights Reserved. 19
Pipeline is a…Ø Groovy DSL for automatic flow description *
• Alternatives – Job DSL, Jenkins Job BuilderØ Job type in Jenkins (Pipeline Job)
• Job definitions can be stored in SCM• Automatic job creation from repositories and organizations
Ø Ecosystem for automation as code• ~20 dedicated plugins• Integration in many other plugins• Suggested by default in Jenkins 2
// Run on a node having the “my_fpga” labelnode("linux && ml509") { git url:"http://github.com/myorg/myproject.git"sh "make all"
} http://bit.ly/pipeline-tutorial
19
© 2017 CloudBees, Inc. All Rights Reserved. 20
Why Jenkins Pipeline?
© 2017 CloudBees, Inc. All Rights Reserved. 21
Why Jenkins Pipeline?
• Robustness - master restarts & network
• Parallelization across multiple nodes:• parallel()• node()
• Ability to store the shared logic in standalone Pipeline libraries
• Security
• Two modes: Scripted and Declarative
• Visualization: Stage View, Blue Ocean
© 2017 CloudBees, Inc. All Rights Reserved. 22
Система
Docs
Plugins Configs
Infra-structure
SamplesDemos
Automation Framework Vision
Infra-structure
Infra-structure
SCM
Infrastructure as Code
Configuration as Code
Documentation as Code
Pipeline Libraries
Pipelines
© 2017 CloudBees, Inc. All Rights Reserved. 23
Developer Experience
• Encapsulation• Simple UI• Most of the code is in
GitHub• No need in classic UI
for 90% of users• Notifications to
email/GitHub
Jenkins Pipeline in Embedded2016
© 2017 CloudBees, Inc. All Rights Reserved. 25
2016
http://bit.ly/nenashev-automotive-day-2016
© 2017 CloudBees, Inc. All Rights Reserved. 26
2016 – Summary
• Jenkins can be used to run tests on hardware in embedded and automotive projects
• Jenkins Pipeline is helpful for test runs• Pipeline lacks some critical features
© 2017 CloudBees, Inc. All Rights Reserved. 27
• Management of resources.• Integration with Throttle Concurrent Builds plugin• Use-case: limiting the license utilization in automation
infrastructures (JENKINS-31801)
• Better support of tools.• Common build flow – dozens of tools• Integration with Custom Tools Plugin seems to be the best option,
especially in the case of multiple tool versions (JENKINS-30680)
• Pipeline package manager (JENKINS-34186)• Lack of plugins for EDA tools in Jenkins, code duplications• A common approach is to keep the shared "functions" in libraries• Pipeline Global Library and Pipeline Remote Loader can be used
2016 – Pipeline improvement areas
From my talk at CloudBees |
Jenkins Automotive and Embedded Day
2016
© 2017 CloudBees, Inc. All Rights Reserved. 28
• Management of resources.• Integration with Throttle Concurrent Builds plugin• Use-case: limiting the license utilization in automation
infrastructures (JENKINS-31801)
• Better support of tools.• Common build flow – dozens of tools• Integration with Custom Tools Plugin seems to be the best option,
especially in the case of multiple tool versions (JENKINS-30680)
• Pipeline package manager (JENKINS-34186)• Lack of plugins for EDA tools in Jenkins, code duplications• A common approach is to keep the shared "functions" in libraries• Pipeline Global Library and Pipeline Remote Loader can be used
2016 – Pipeline improvement areas
From my talk at CloudBees |
Jenkins Automotive and Embedded Day
2016
© 2017 CloudBees, Inc. All Rights Reserved. 29
Management of resources• Lockable Resources Plugin – since 2016• Throttle Concurrent Builds – coming soon• Pipeline Milestones
© 2017 CloudBees, Inc. All Rights Reserved. 30
PARTIAL - Better support of tools
• withEnv() for environment variables• tool() step in Pipeline was improved
• Tool reference in the Job• Section in Declarative Pipeline
• Custom tools is not fully integrated• Follow JENKINS-30680• ETA – Q2, 2017
© 2017 CloudBees, Inc. All Rights Reserved. 31
DONE – Pipeline Libs (Pipeline Package Manager)
• Early 2016 • Global Pipeline Library• Pipeline Remote Loader Plugin
• Late 2016• Pipeline Shared Libraries• Support of @Grab – Pipeline libs
can pull common Groovy libs
Library Structure:
© 2017 CloudBees, Inc. All Rights Reserved. 32
• Pipeline Shared Libraries• Libs are located in SCM• Versioning• Definition for scopes:▸ Jenkins system-wide▸Folder▸ Job
© 2017 CloudBees, Inc. All Rights Reserved. 33
Pipeline Library Definition [1/2]
© 2017 CloudBees, Inc. All Rights Reserved. 34
Pipeline Library Definition [2/2]
© 2017 CloudBees, Inc. All Rights Reserved. 35
Using Pipeline Libraries
• Implicit Loading• Via Jenkins Global or Folder configuration
• Explicit Loading• @Library
© 2017 CloudBees, Inc. All Rights Reserved. 36
Using Pipeline Libraries
com.mycorp.libs.make
Jenkinsfile
makeProject()
makeProject() implementation - global variable
com.mycorp.libs.coverity
com.mycorp.libs.cppunit
Globally-defined library, automatic
update
@Library() definition
© 2017 CloudBees, Inc. All Rights Reserved. 37
Early adoption of any framework
http://devopsreactions.tumblr.com/post/150674592321/petting-your-incomprehensible-piece-of-code
• It is working!• In several pet projects…
Lessons Learnt.What does NOT work well?
Lessons Learnt.What does NOT work well?
• Development Tools• Non-Success Path handling• Declarative Pipeline• Manual release management
© 2017 CloudBees, Inc. All Rights Reserved. 40
Pipeline Developer Experience
• No Debugger• No IDE• No built-in test framework• Limited static analysis features
•Pipeline DSL != Groovy
© 2017 CloudBees, Inc. All Rights Reserved. 41
How do I work?
Intellij IDEAFilesystem SCM Plugin (with patch)
Disclaimer: There may be better ways
• Syntax file• Built-in documentation
Jenkins Test Instance
Source Code (local .git repos)
• Production System Snapshot
• Custom Pipeline Library Settings
• Repositories for libraries• Jenkinsfile repos
© 2017 CloudBees, Inc. All Rights Reserved. 42
Getting the Syntax File
© 2017 CloudBees, Inc. All Rights Reserved. 43
Filesystem SCM Plugin
• Pending pull request:• https://github.com/jenkinsci/filesystem_scm-plugin/pull/2
Library definition in the test instance
!!!
© 2017 CloudBees, Inc. All Rights Reserved. 44
Pipeline Testing
• No standard test framework offered by the Jenkins project• External project: https://github.com/lesfurets/JenkinsPipelineUnit
“mvn verify” has been invoked…
And the execution passed…
For the source Jenkinsfile…
Lessons Learnt.What does NOT work well?
• Development Tools• Non-Success Path handling• Declarative Pipeline• Manual release management
© 2017 CloudBees, Inc. All Rights Reserved. 46
Error Handling
• IMHO: Pipeline is designed for Success Path scenarios
When it “Just works”
When it does not
© 2017 CloudBees, Inc. All Rights Reserved. 47
Error Handling. Top-levelScripted Pipeline Declarative Pipeline
• Global try/catch()
When it does not…
© 2017 CloudBees, Inc. All Rights Reserved. 48
Error Handling
When it “Just works”
When it does not
The interesting part…Failover, repeat
© 2017 CloudBees, Inc. All Rights Reserved. 49
Error Handling. Hardware failover
• Pipeline allows re-running unreliable stages on other nodes.
System failure(script returns 253)
Node #1 Node #2. . .
. . .
for (def board : boards) {echo "trying board " + board;try {node(board) {checkout scmsh ‘./bin/run.sh’ // Call passed => DONEbreak;
}} catch (Exception ex) {if (ex.message.contains
("exit code 253")) {// Fatal errorfail("Test run failed")
}}
}
Try all possible boards:
© 2017 CloudBees, Inc. All Rights Reserved. 50
Error handling. Hardware failover
for (def board : boards) {echo "trying board " + board;try {node(board) {checkout scmsh ‘./bin/run.sh’ // Call passed => DONEbreak;
}} catch (Exception ex) {if (ex.message.contains
("exit code 255")) {// Fatal errorfail("Test run failed")
} } }
???
• Hard to extract error cause• Special handling is required for
every case• Cannot just use existing parsers
• Build Failure Analyzer• Log Parser• Warnings
• Try-catch blocks pollute the code• Hard to test
© 2017 CloudBees, Inc. All Rights Reserved. 51
Context retention
• No way to propagate any data• Solution - @NonCPS steps within the library steps
© 2017 CloudBees, Inc. All Rights Reserved. 52
Error handling. Hardware failover
Declarative Pipeline
Encapsulation via Libraries
Simple use-case
Other
• Exception try/catch• timeout() wrapper• Proper failover• Logic in @NonCPS
Lessons Learnt.What does NOT work well?
• Development Tools• Non-Success Path handling• Declarative Pipeline• Manual release management
© 2017 CloudBees, Inc. All Rights Reserved. 54
Declarative Pipeline
• It is great for simple use-cases• Publishers & Notifiers
• We do not need a framework for simple use-cases
• Lack of features• “How do I pass option for the Docker Agent”• “How do I declare a resource requirement for Throttling?”• …
• Scripted Pipeline is a recommended fallback, but there is no automatic migration
© 2017 CloudBees, Inc. All Rights Reserved. 55
Declarative Pipeline. Recommendations
• Use Declarative Pipeline with caution• Complex agent definitions – use Scripted Pipeline• Complex throttling – use Scripted Pipeline• Post-build steps – work well• Tool definitions – work well• …
Lessons Learnt.What does NOT work well?
• Development Tools• Non-Success Path handling• Declarative Pipeline• Manual release management
© 2017 CloudBees, Inc. All Rights Reserved. 57
Pipeline vs. Application Lifecycle
© 2017 CloudBees, Inc. All Rights Reserved. 58
Pipeline vs. Application Lifecycle
BUT:• Non-CD process with manual promotion• Dependent products• Snowflake development stages
© 2017 CloudBees, Inc. All Rights Reserved. 59
Solutions
• Input() step + milestone() for synchronization• Multiple Pipelines
• Release() step is coming soon• Release Plugin
Pipeline 1 Pipeline 2Run Selector
Staging Artifact Storage
External ALM tool
CD for the Component 1
CI/CD for the Component 2
Lessons learnt• No area-specific issues so far• High barrier to entry• It worth investment
© 2017 CloudBees, Inc. All Rights Reserved. 61
What I need? 2017
• Development Tools for Pipeline• IDE, Debugger, Test Tools
• Community Pipeline Libraries• Common Embedded cases without plugins
• Promotion engine for Pipeline• Ideally: Merge of standard Promoted Builds and Release plugins
• Plugins for area-specific tools
© 2017 CloudBees, Inc. All Rights Reserved. 62
Takeaways
• Jenkins Pipeline is a powerful engine for highly-customized Jenkins instances
• Jenkins Pipeline can be effectively used in Embedded/Automotive use-cases
• Barrier to entry is still high
© 2017 CloudBees, Inc. All Rights Reserved. 63
More info
• Embedded solutions page:• https://jenkins.io/solutions/embedded/
• Links to the articles and videos• Some categorization
Embedded Solutions Page
…
© 2017 CloudBees, Inc. All Rights Reserved. 64
• Jenkins project:• Website: https://jenkins.io• Embedded Solutions page: https://jenkins.io/solutions/embedded/
• Pipeline:• Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md• Compatibility: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md• Examples: https://github.com/jenkinsci/pipeline-examples
• CloudBees website:• Website: https://www.cloudbees.com• CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform
Useful Links
© 2017 CloudBees, Inc. All Rights Reserved. 65
Thank you!
Contacts:
E-mail: onenashev@cloudbees.com
GitHub: oleg-nenashev
Twitter: @oleg_nenashev
Software at the speed of ideas
THANK YOU!www.cloudbees.com