Download - Continuous Integration and Deployment using OpenStack

Continuous Integration and Deployment using OpenStack

May 5, 2014

Miguel Zuniga@[email protected]

eBay Inc. enables commerce by delivering flexible and scalable solutions that foster merchant growth.

About ebay inc

With 145 million active buyers globally, eBay is one of the world's largest online marketplaces, where practically anyone can buy and sell practically anything.

With 148 million registered accounts in 193 markets and 26 currencies around the world, PayPal enables global commerce, processing almost 8 million payments every day.

eBay Enterprise is a leading provider of commerce technologies, omnichannel operations and marketing solutions. It serves 1000 retailers and brands.

3

Agenda

• Enter Fluo (CI/CD)• Fluo Flow• Fluo Code Replication• Packages, Artifacts, <your term here>• Distribution of Packages• Infrastructure as Code• Deployment to Production• Screenshots• Roadmap


Enter Fluo (CI/CD)

From begin to end… how code goes to deployment.

5

At the beginning

Basic system running• Gerrit• Jenkins• Web server

Requirements

• Single Interface• Scalable• Simple to use• Developer Friendly• Generic


6

Fluo and the CICD configuration file

• What is Fluo?– WebUI and API– Single pane of glass– Cloud instance prov and decomm– Allows user’s to configure the system– RBAC

• How it works?– Jenkins through API– Gerrit through API– Uses ZMQ to communicate with workers which do non-API actions– Recycle slaves or provisions them on the fly at the moment of job execution.

• What is the cicd configuration file?– Specification and requirements files– How the developers instruct the system what to do (run unittests, how to build packages, install

dependencies, more…).

• Example


7

Fluo cicd configuration file

language: bashpackage_install: apt: - build-essential - ruby1.9 - ruby1.9-dev - mysql-server - mysql-dev gem: - rails -v=4.0.3 - zmq - mysql2before_review: - /etc/init.d/mysql-server start - mysql -u root < database.sql - RAILS_ENV=test rake db:migrate - RAILS_ENV=test rake db:test:loadreview_script: - rake test test/models/user_test.rb - rake test test/models/session_test.rb - rake test test/models/post_test.rbnotification: email: - [email protected]


8

Fluo


Components Architecture

• Fluo (App/worker/coordinator)• Cloud for provisioning

instances• Gerrit• Jenkins• Zuul• Galera (MySQL)• Zabbix (or any other

monitoring)• Puppet (or any other CM tool)• Mrepo• Rsync

Fluo Flow

A step by step journey of a code change.

10

Basic Workflow


• Fluo basic workflow has 6 different stages/step– Review– Approval– Build– Integration– Release– Periodic

• At each stage/step Fluo will:1. Build a cloud instance2. Read the configuration file 3. Execute the scripts defined4. Report back status5. Destroy the instance

• Customs workflows can be created by users.

Developer FluoCommit and request review

Execute Review Scripts

Approve the code change Execute Approval ScriptsMerge and replicate code

Execute Build Scripts

Adds “Run Integration” to comment history on specific change

Execute Integration Scripts

Tag’s a specific commit Execute Release Scripts

At XY time of day Execute Periodic Scripts

11

Code Quality = Must have… Review, Approval, Build


The rest of the flow… Integration… Release… Periodic.

Review• Code change

submitted for review

Approval• Executed before

merging the code when code change is approved by someone

Build• Executed after the

code change has been merged

Integration• Activated by user

through Fluo by commenting “Run Integration” on a specific code change.

Release• Activated by user

adding tags to a specific commit.

Periodic• Activated by at a

specific time of the day, week. Similar to a cron job.

Fluo Code Replication

One project… 2 projects… Multiple projects.. Best Practices.

13

Github, Internal Github and your own Git repository.

Some points to consider:

• Categorize your code– Will it be open sourced?– Is it private for our team?– Is it private for our company?– Do we need mirrors?

• Always use ssh keys

• Standards– User which will be replicating– Replicate Branches or maybe not…


Fluo

Github

Local Github

Private Github

Packages, Artifacts, <your term here>

How do you want to wrap it out? RPM? Debs? Tar?

15

When and where do I create my packages?

Things to consider.• Jenkins slaves will build the packages.• Make sure all your dependencies are in your cicd config file.• Prepare your environment with the package_install and before_{stage} sections.• Build your package at the {stage}_script section.• Define a versioning standard (0.0.x for test packages, x.y.0 for production packages)

Two main stages were created for build purposes.• Build stage

– Use commit number + time in secs as package version.• Release stage

– Use git tag as package version.


Distribution of Packages

Shipping the code to different locations.

17

Shipping Packages to a Central Point

• Use simple and secure methods.– Secure Object storage– Rsync + SSH– SCP

• Mrepo (if you are RPM Based)• Replication is done every 5 mins.


Replicate… Replicate and Replicate.

Fluo Package

Repository

Cloud A

Datacenter 2

Cloud B

Datacenter 1

Infrastructure as Code

Controlling and Managing what is your Infrastructure doing.

19

Configuration Management

• At the moment we use Puppet.• Puppet Code will also go through Fluo.


Some guidelines to our puppet coders.

• Create your CM code in a service oriented way.• Use virtual resources.• Create within a module sections for each type

of OS you work with.• Create a common module.• Use parameterized classes.• Puppet parser validate filename.pp at the

review/approval stage.• Puppet apply –vd --noop

--modulepath=/modules filename.pp at the review/approval stage.

Example:• virtual.pp• classes

• someclass.pp • modules

• Fluo• files• templates• manifests

• init.pp• Redhat

• install.pp• config.pp• service.pp

• Ubuntu• install.pp• config.pp• postconfig.pp• service.pp

Deployment to Production

So it’s GO time and you have green light.

21

It’s GO time

• Centralize your configuration management code

• Keep it in sync (pull from the repository every 2 or 5 mins).

• Tag your Releases with a chronological meaning and keep a standard– OK

• 1.1.0• 1.2.0• 1.10.0

– NOT OK• V1.0• Beta1.0.1• 1.2

• Remember the TAG goes into the name of the package and its how your package manager identifies which package should be installed on an upgrade.

• Automatic Deployment or Scheduled Deployment?


Some Fluo screens

23Continuous Integration and Deployment using OpenStack

Roadmap

28

The Future of Fluo at eBay Inc.

• CICDaaS (CICD as a Service).• Integrate with other CM tools (chef, salt).• Completely control of CM tools through Fluo.• Container support (OpenVZ/Docker).• Add block storage as option to store packages/artifacts.• Ironic, Cobbler, Foreman, Razor integration.• Jenkins? Zuul? • Improve UI.


Thank you.

Yes. We are hiring!

[email protected]