JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects
-
Upload
cloudbees -
Category
Technology
-
view
137 -
download
0
Transcript of JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects
![Page 1: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/1.jpg)
Jenkins World Tour 2015 London, UK 23-24 June
![Page 2: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/2.jpg)
Footer
#jenkinsconf
Jenkins-based CI for heterogeneous Hardware/Software projects
Oleg Nenashev Jenkins Developer @ CloudBees
2
![Page 3: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/3.jpg)
Footer
#jenkinsconf > whoami • https://ru.linkedin.com/in/onenashev • Background: ~10 years of…
– R&D: HW and Embedded systems – CI in HW, Embedded and EDA areas – Department-wide CI frameworks
• Current: – Jenkins CI contributor – Jenkins Developer at CloudBees – PhD student, EDA Lab @ SPBPU
3
![Page 4: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/4.jpg)
Footer
#jenkinsconf > whoami --list jenkins • >6 years of Hudson/Jenkins experience • 3 years of open-source contribution to Jenkins • Maintenance of large-scale installations • I maintain about 10 open-source plugins • Contributions to other Jenkins stuff and INFRA
• Ownership • Job Restrictions • Role Strategy (legacy)
• Custom Tools (legacy) • Extra Tool Installers • Cygwin Process Killer
Tool belts Ownership-based security
![Page 5: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/5.jpg)
Footer
#jenkinsconf Outline
• CI in HW/Embedded world. Why Jenkins? • Integrating Jenkins with the common stuff
– EDA tools, FPGA farms, Computing Grids • Several examples and case studies • Getting added value from CloudBees Jenkins
Platform
5
![Page 6: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/6.jpg)
Footer
#jenkinsconf Jenkins users from HW world
• The most of HW companies have Jenkins instances • Examples from a 5min search in public materials
6
![Page 7: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/7.jpg)
Footer
#jenkinsconf Related talks / articles • Robert Martin, “Multi-Stage-CI with Jenkins in an Embedded
World” @ JUC2014/Berlin • Martin d’Anjou, “FPGA Continuous Integration with Jenkins”
@SNUG2013/Canada • Alan Fitch, “Continuous Integration for FPGA Design and
Verification” @ Verification Futures, 2015 • JL Gray, Gordon McGregor, “A 30 Minute Project Makeover
Using Continuous Integration” @ DVCon, 2012 • …
• Sarah Woodall, “The Famous Cows of Cambridge: A Non-Standard Use Case for Jenkins” @JUC2015/London
![Page 8: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/8.jpg)
Footer
#jenkinsconf
INTRODUCTION CI in HW/Embedded world
8
![Page 9: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/9.jpg)
Footer
#jenkinsconf What is a hardware product?
. . .
Not so obvious… • IP Libraries –
reusable blocks • Prototypes • Development kits • …
![Page 10: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/10.jpg)
Footer
#jenkinsconf HW Development Flow
10
Synthesis
Place & Route
RTL Design
Specification
= Compilation
= Linking
Design
= Coding
VHDL Verilog / SystemVerilog SystemC …
Verification
Packaging
![Page 11: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/11.jpg)
Footer
#jenkinsconf HW CI Example
11
Get sources SCMs Binary repositories Build artifacts
Setup environment
“Build” = S&P&R
“Test”
“Analyze”
Publish reports
Deploy artifacts
Static analysis Formal verification Timing analysis
Specialized tools
Unit tests Integration tests
EDA Tools Electronic Design Automation
Test results Coverages System resources utilization
Much parallel tasks in flows
![Page 12: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/12.jpg)
Footer
#jenkinsconf Embedded SW build process
12
Other software
Embedded Software
• Different tools, but the approaches are similar • Verification is a real challenge
![Page 13: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/13.jpg)
Footer
#jenkinsconf And what about an HW IP Product?
Hardware part
External IP Firmware / Embedded
SW
External SW
Development tools / EDAs
Demo SW
Demo hardware
Developer kits
Yes, we need to automate everything
“Main product”
![Page 14: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/14.jpg)
Footer
#jenkinsconf Welcome to CI hell! • Almost everything needs to be automated • Multiple technologies and tools • Unstable hardware & networks • Legacy stuff is everywhere • HW engineers are not SW experts • … • Limited resources • Limited OSS usage and contribution options
![Page 15: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/15.jpg)
Footer
#jenkinsconf Why Jenkins?
• It’s a framework => you can do everything • It’s open-source:
– You can customize Jenkins core and plugins – Plenty of available materials and examples
• “Stable” LTS versions • Commercial support for self-hosted installations
![Page 16: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/16.jpg)
Footer
#jenkinsconf
AREA SPECIFICS
• EDA Software • Computing grids • FPGAs
16
![Page 17: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/17.jpg)
Footer
#jenkinsconf EDA Software
• Almost no OSS plugins – Exception - Cadence vManager Plugin
• Specifics: – Dozens of tools in a common design flow – Different platform requirements – Multiple tool versions are required – Big size of tools – up to 100 Gigabytes – Limited licenses being shared with externals
17
![Page 18: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/18.jpg)
Footer
#jenkinsconf EDA Software. Possible solution
• Store tools on shared folders – On Windows services kohsuke/winsw/pull/71 may be used – Extra Tool Installers Plugin provide a stub installer
• Access tools: Labels + Custom Tools Plugin • Idea of the plugin:
– Configurable installers – Setup of a custom environment (PATH, env. vars) – Built-in versioning based on Extended Choice Parameter
18
![Page 19: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/19.jpg)
Footer
#jenkinsconf Custom Tools. Global Configuration
19
Can be stored remotely => automatic tool deployment
Node and global env. vars are supported
![Page 20: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/20.jpg)
Footer
#jenkinsconf Custom Tools. Job Configuration
20
Allow versioning
Automatic installation
![Page 21: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/21.jpg)
Footer
#jenkinsconf Custom Tools. Usage
21
In build steps:
In build launch dialog:
![Page 22: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/22.jpg)
Footer
#jenkinsconf EDA Software. Publishers
• Jenkins has publishers for many SW dev. tools – Unit test results (xUnit et all.) – Performance test results – Coverages
• No solutions for hardware, but you can use converters • Example for VCS coverages:
1. Get an HTML or raw XML report 2. Use XSLT to convert it to Cobertura format 3. Publish data using a standard plugin
22
![Page 23: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/23.jpg)
Footer
#jenkinsconf EDA Software. Licenses • FLEX License Manager is everywhere in EDA • Main problem: shared usage of licenses
– No sporadic prioritization in FLEX => long waiting – Dedicated licenses are not cost-effective – Limited availability of tools => custom accounts
• Minimal solution – Throttling, custom “wait for a license” scripts
• Best solution - FLEX LM plugin, no OSS ones
23
![Page 24: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/24.jpg)
Footer
#jenkinsconf Hardware testing
• Simulators – specialized EDA tools
• Hardware – FPGAs, ASIC prototypes
• Combined solutions (HW Accelerators, etc.)
24
![Page 25: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/25.jpg)
Footer
#jenkinsconf Cloudless skies
• Simulation tests require…
• Public/private clouds are not cost-effective • HW Companies mostly use computing grids
25
Thousands CPU hours
Terabytes of TMP files
and reports All RAM you have
![Page 26: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/26.jpg)
Footer
#jenkinsconf Computing grids
Examples: • Sun Grid Engine (SGE, UGE) • Platform Load Sharing Facility (LSF) • NetBatch @ Intel • …
26
• Datacenter-wide grids • Sharing across
departments “We run hello-world in 2 hours”
![Page 27: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/27.jpg)
Footer
#jenkinsconf Computing Grids Usage
27
Manage tasks from CLI (script steps)
“Transparent” – launch Jenkins nodes
on the grid
• Use-case: Legacy scripts • Difficult management (job
abort, status polling, etc.)
• Manual launch • “Start by script on Master” • LSF Cloud Plugin • No FOSS plugins for
other grids L
“Launch task” build & workflow steps • No FOSS plugins !
![Page 28: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/28.jpg)
Footer
#jenkinsconf Simulations are slow, so…
28
• S - EZ1CUSB @ http://www.easyfpga.com • M - http://www.xilinx.com/univ/xupv5-lx110t.htm • L - HAPS-64, www.synopsys.com
S - x00$ M – x000$
L –x0000$
FPGA accelerators & prototyping boards
![Page 29: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/29.jpg)
Footer
#jenkinsconf
29
And XXL…
• http://blogs.synopsys.com/breakingthethreelaws/2014/04/the-secret-ninja-fu-for-higher-performance-prototype-operation/
HAPS S96
Guess the price… Could CI have its own stuff?
Your CPU prototype is here
![Page 30: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/30.jpg)
Footer
#jenkinsconf Mapping CI to existing HW
30
Nodes @ Site A Nodes @ Site C Nodes @ Site B “We borrowed FPGAs here, somebody takes them randomly”
“This stuff has Windows drivers only”
“No infra team members here”
“We run nodes on dev. laptops”
Frankenboards “We built it from the garbage we had”
Infrastructure after the “budget optimization”
L
L
L
L
L
![Page 31: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/31.jpg)
Footer
#jenkinsconf How it works
31
Power
When it hangs
“Deploy bit-files” Blasting cables
Debugging cables, firmware upload
We don’t change
them, huh?
Jenkins node
drivers
tools
peripherals
Attached hardware
runtime
![Page 32: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/32.jpg)
Footer
#jenkinsconf How do I implement jobs?
• Throttling of builds • Shared functionality is decoupled to external project
(Parameterized Trigger or Template Project) – FPGA blasting – Firmware Upload – Run of a test case (binary upload) using a debugger provided
by the toolchain (gdb, etc.) – etc.
• Failover and restart on infra issues => Naginator
32
![Page 33: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/33.jpg)
Footer
#jenkinsconf Throttling of FPGA jobs • Throttle Concurrent Builds Plugin
– Jobs can be throttled by a category – Performance issues (esp. till 1.609.1), can be partially solved
by Label Throttle plugin (CloudBees Jenkins Enterprise) – Job developers must follow rules, no enforcement
Global configuration:
33
![Page 34: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/34.jpg)
Footer
#jenkinsconf Throttling of FPGA jobs
Job configuration:
34
![Page 35: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/35.jpg)
Footer
#jenkinsconf FPGAs monitoring
• Impossible to use an external system – shared resources • Solution in Jenkins:
1. Use periodic Multi-configuration Matrix Job for a monitoring 2. Use Elastic Axis to generate label axes by an expression 3. Run a spot-check on FPGA with blasting and a firmware test 4. If it fails, disable the node using System Groovy Script 5. Send notifications to INFRA admins
35
![Page 36: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/36.jpg)
Footer
#jenkinsconf
ENTERPRISE FEATURES Getting added value from CloudBees Jenkins Platform
36
CJE – CloudBees Jenkins Enterprise CJOC – CloudBees Jenkins Operations Center
![Page 37: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/37.jpg)
Footer
#jenkinsconf Common features All common features are
applicable, out of the scope See CloudBees Docs
![Page 38: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/38.jpg)
Footer
#jenkinsconf Templates!
• OSS Plugins have a limited functionality: – Template Project, Inheritance Plugin, Job Generator, etc.
• Parameterized trigger causes UI overheads – Multi-page configuration in UI: “Trigger with several
parameters => wait => copy artifacts => analyze” • Templates allow to save much time on the
implementations • Workflow + Scriptler Plugin are tasty as well
38
![Page 39: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/39.jpg)
Footer
#jenkinsconf Easy failover implementation • Workflow allows to transparently re-run
unreliable stages on other nodes. for (def board : boards) { try { // blasting …
// run test …
// test finished break; } catch (FPGAError error) { // try another board continue; } }
System failure
Node #1 Node #2 . . .
. . .
![Page 40: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/40.jpg)
Footer
#jenkinsconf Shared slaves for rare hardware • Use-case: multiple Jenkins instances in a single org • Instances may require a specific resource • No cross-Jenkins throttling features • Out-of-the-box throttling in CJOC shared slaves
40
![Page 41: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/41.jpg)
Footer
#jenkinsconf Durable tasks
• Endurance tests may take weeks • Jenkins slaves may be disconnected, masters can
be restarted for maintenance purposes • No stable solutions in Jenkins OSS if you don’t use
computing grids • Durable tasks allow to overcome the issue in many
cases – Shell scripts are running independently – Jenkins polls for a completion asynchronously – Serious limitation: cross-site installations
41
![Page 42: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/42.jpg)
Footer
#jenkinsconf Summary
• Jenkins is not only about software
• Jenkins can be used to run CI in hardware projects
• You can use Jenkins even without domain-specific plugins
• CJE/CJOC features greatly help to decrease the maintenance cost
42
![Page 43: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/43.jpg)
Footer
#jenkinsconf My HW-related wish-list
• Sun Grid Engine Cloud plugin • FLEX LM plugin • Flexible Windows Services
• Plugins for common EDA tools and report formats – Build steps are OK => just use CLI utilities – Publishing requires much efforts – Logs parsing would be a killer feature
43
![Page 44: JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects](https://reader031.fdocuments.in/reader031/viewer/2022022205/58cef4991a28abab738b4bc9/html5/thumbnails/44.jpg)
Footer
#jenkinsconf
Q&A