Software Factory: Continuous Integration

Software Factory: Continuous Integration and Delivery, OpenStack-Style

http://softwarefactory-project.io

Freenode: #softwarefactory

Fabien BoucherMatthieu HuinOctober 28, 2015

http://softwarefactory-project.io/

Continuous Integration and Continuous Delivery: A State of the Art

Software Factory: CI and CD, OpenStack-style4

Typical Workflow


Problems

With the entry price:● Cost of automation● Cost of dedicated CI infrastructure

With the workflow itself:● commits in rapid succession can make CI tests obsolete as they run● It is possible to end up with a broken master temporarily (and not even knowing it)


Conventional CI workflows don't scale to large open source projectsOpenstack is … [*]

● 43 official projects, each with its own subset of deliverables, and most of them strongly tied to each other in deployment

● More than 2000 contributors● 70K+ patchsets submitted in the last development cycle, or roughly 400 patches a

day• Some patchsets can trigger up to 25 jobs● Some jobs can take more than 30 minutes to execute

[*] Ballpark estimations from the Openstack governance project and stackalytics for the Liberty development cycle


The OpenStack Way


The OpenStack Infrastructure

Local changes submitted via

git-review

Gerrit code review Zuul

Jenkins (gearman-plugin)

pipy mirrors & other semi-privileged servers

Git repository

Jenkins01,02...

Github mirror git.openstack.org mirror

Gearman Server

slave nodes managed

by Nodepool


Problems SolvedAdvantages Drawbacks● Guarantee that the master version is

always OK at any time● Infrastructure costs are reduced with

Nodepool● Gated commits with Zuul ensure that

tests are never obsolete

● Longer workflow● Extra workload on core developers

… and all the deployment recipes for OpenStack's CI infrastructure are freely available !

https://git.openstack.org/cgit/openstack-infra/


Can I Have One Too ??

Maybe, but with a lot of work to adapt OpenStack's recipes

https://git.openstack.org/cgit/openstack-infra/

Software Factory, the OpenStack-like CI with batteries included


Main Features

● All the good stuf from the Openstack CI and more, and ready to use!● Easy to deploy and update● The projects configuration is versioned and follows a Continuous Deployment

workflow● Comes with template jobs, like artifacts export to Swift (object storage)


Core Elements

ManageSF Dashboard

Cauth SSO

Redmine Ticketing System

& Backlog

Jenkins Job Manager

Etherpad Collaborative

Text Editor

Zuul Master of the Gate

Nodepool Slave Manager

Pastie Quick Text Paste

Gerrit Code Review

GitWeb Code Repository

Slave Nodes

Web UI

REST API

Clone project

Review

Workflow

Hooks

Authentication


Code Review with Gerrit

● Preconfigured to enable a review workflow inspired by Openstack:● Preconfigured labels (Code Review/Verified/Workflow)● Preconfigured ACLs mapping users to labels types

● Scoring by label generates events and triggers actions along the CI pipeline● Integration with the ticketing system


Gating with Zuul: Continuous Integration

● Zuul comes preconfigured to work with Gerrit and Jenkins within Software Factory

● Integration pipelines:● check: jobs executed after proposing a patch● gate: jobs executed prior to the merge (Gated commit)


Gating with Zuul: Continuous Deployment

● Delivery and Deployment pipelines:● post: jobs executed after the merge● tag: jobs executed after pushing a tag● periodic: jobs executed on a daily basis

● Credentials management in Jenkins


Slave Nodes Management

Two possibilities in Software Factory:

● Managing static, persistent slaves● Letting Nodepool manage a dynamic pool of image-based slaves in an Openstack

Cloud for you


Benefits of Using Nodepool

● Jobs scalability● Smarter resource usage● Improved Jobs stability and reproducibility compared to long-lived slaves


Collaborative Tools

ManageSF Dashboard

Cauth SSO


& Backlog

Jenkins Job Manager


Text Editor




Gerrit Code Review


Slave Nodes

Web UI

REST API

Clone project

Review

Workflow

Hooks

Authentication


Keeping it all together

ManageSF Dashboard

Cauth SSO


& Backlog

Jenkins Job Manager


Text Editor




Gerrit Code Review


Slave Nodes

Web UI

REST API

Clone project

Review

Workflow

Hooks

Authentication


Deployment as easy as 1, 2, 3 ... 4

1. Fetch the pre-built QCOW2 image2. Adapt the unified configuration file sfconfig.yaml3. Run the configuration script4. And you are ready to host projects and perform CI/CD on them "à la OpenStack"


The Config Repository

● Manages Jenkins jobs, Zuul layout, Nodepool images/labels/scripts● Users can propose changes via Gerrit● Follows the CI/CD workflow● Any proposed changes are validated then merged by pre-defined jobs "config-

check" and "config-update"

Development Workflow within Software Factory


A View of the Dashboard


Creating a Project


Adding Users to the Project


Configuring Slaves with Nodepool (1)Images configuration by users, stored in the config repository

- provider: defaultimages: - name: bare-centos-7 base-image: CentOS-7-cloud username: centos private-key: /var/lib/jenkins/.ssh/id_rsa setup: base.sh min-ram: 2048


Configuring Slaves with Nodepool (2)Labels configuration in the config repository

labels:- name: bare-centos-7 image: bare-centos-7 min-ready: 1 providers: - name: default


Configuring Jobs for a ProjectJenkins Job Builder template language

- job: name: 'myproject-functional-tests' defaults: global builders: - prepare-workspace - shell: cd $ZUUL_PROJECT && DEBUG=1 ./run_functional-tests.sh triggers: - zuul publishers: - fetch-my-artifacts - zuul-swift-upload node: bare-centos-7


Configuring Jobs for a ProjectZuul Pipelines Configuration

- name: myproject check: - myproject-functional-tests gate: - myproject-functional-tests - myproject-upgrade-tests


Issue Tracking


Code Review


Gating and Job Parallelization


Checking Tests Results (1)


Checking Tests Results (2)


Delivering the GoodsDeployment job requiring credentials

- job: name: 'myproject-publisher' defaults: global builders: - prepare-workspace - shell: | rsync -a --delete -e "ssh -i $key" $ZUUL_PROJECT [email protected]:. ssh -i $key [email protected] $ZUUL_PROJECT/deploy.py triggers: - zuul wrappers: - credentials-binding: - file: credential-id: 0f3ff035-90df-4f6e-84fb-e924c26e4ab0 variable: key node: 'publisher'

- name: myproject post: - myproject-publisher

THANK YOUhttp://softwarefactory-project.io

Freenode: #softwarefactory

ML: [email protected] / [email protected]

http://softwarefactory-project.io/

mailto:[email protected]

mailto:[email protected]

Software Factory: Continuous Integration

Documents

Transcript of Software Factory: Continuous Integration