Contributing to OpenStack

Tutorial on Contribu-ng to OpenStack

Devda8a Kulkarni PTL, OpenStack Solum

Agenda

•  Part I –  Introduc-ons –  Introduc-on to OpenStack

•  Part II –  Deep dive of a OpenStack project -‐ Solum

•  Part III –  Contribu-ng to OpenStack –  OpenStack CI system –  Development best prac-ces –  Hands-‐on exercise

•  Ques-ons and Answers

Agenda

•  Part I –  Introduc,ons –  Introduc,on to OpenStack




Introduc-on

•  Me – Devda8a Kulkarni – First contributor on Solum – Currently Project Team Lead (PTL) of Solum for Mitaka cycle

– PhD, University of Minnesota Minneapolis

•  You

Introduc-on to OpenStack

What is OpenStack?

A family of “open” projects that together enable different aspects of cloud compu-ng Openness of OpenStack h8ps://wiki.openstack.org/wiki/Open

•  Open source •  Open design •  Open development •  Open community

OpenStack family

•  Family of projects enabling different aspects of cloud compu-ng –  Authen-ca-on and authoriza-on layer

•  Keystone –  Infrastructure-‐as-‐a-‐service layer

•  Nova, Glance –  Storage layer

•  Swi[, Cinder –  Networking layer

•  Neutron –  Pla\orm-‐as-‐a-‐service layer

•  Solum, Heat, Murano, Mistral, Trove

•  Project types –  OpenStack big tent vs. OpenStack related projects

OpenStack by numbers

•  49 projects in OpenStack big tent •  Several hundred organiza-ons involved •  Several thousand developers par-cipa-ng

Agenda





What is Solum?

OpenStack project that provides easy mechanism for applica-on developers to deploy and run their applica-ons on OpenStack star-ng from applica-on’s source code Why should you care? (the value proposi-on)

–  For operators, Solum provides ability to make your OpenStack cloud more useful for your clouds’ applica-on developers by simplifying the process of deploying applica-ons to it

–  For developers, Solum provides an easy-‐to-‐use pla\orm for building, tes-ng, and deploying applica-ons on OpenStack clouds

Project goals

Applica-on Stack Flexibility Ability to support applica-ons wri8en in different languages and using different applica-on frameworks

Developer Produc-vity Ability to perform CI/CD, integra-on with github Add-‐On Services Extensibility Ability to support different add-‐on services

Applica-on Portability Ability to deploy applica-ons across different OpenStack clouds (use na-ve OpenStack services)

Project details

•  Wiki: h8ps://wiki.openstack.org/wiki/Solum

•  Mee-ngs: Tuesdays 1700 UTC in #openstack-‐mee-ng-‐3

•  IRC: Channel: #solum on freenode

•  Code: h8ps://github.com/openstack/solum h8ps://github.com/openstack/python-‐solumclient

Solum abstrac-ons

•  App •  Languagepack (LP) •  Deployment Unit (DU) •  Workflow •  Add-‐on

Solum abstrac-ons -‐ App

Declara-ve applica-on defini-on

•  source code repository •  run command •  port •  languagepack

version: 1 name: cherrypy description: python web app languagepack: python source: repository: https://github.com/rackspace-solum-samples/solum-python-sample-app.git revision: master workflow_config: run_cmd: python app.py ports: - 80

Solum abstrac-ons -‐ Languagepack (LP)

•  Docker image which contains applica-on specific build and run-me libraries

•  A LP needs to implement Solum’s languagepack contract A well-‐known file available at a known loca-on on the languagepack image /solum/bin/build.sh

•  How to build a LP? Dockerfile

•  Operator-‐defined or user-‐defined

Solum abstrac-ons -‐ Languagepack Example: Python

FROM ubuntu:precise MAINTAINER Murali Allada <[email protected]> RUN apt-get -yqq update RUN apt-get -yqq install python-pip RUN apt-get -yqq install python-dev COPY build.sh /solum/bin/

https://github.com/rackspace-solum-samples/solum-languagepack-python

#!/bin/bash # Check if pip is installed pip help [[ $? != 0 ]] && echo python-pip is not installed. && exit 1 # Install app dependencies cd /app pip install -r requirements.txt

build.shDockerfile

Solum abstrac-ons -‐ Deployment Unit (DU) (1/2)

•  Docker image that is formed from the languagepack image with applica-on’s source code added to it

DU = LP + applica-on source code

•  Solum guarantees the DU contract Applica-on source code available at a known loca-on on the DU image /app

•  How to build a DU? –  Construct Dockerfile with languagepack as the base image, applica-on source code

injected, run command as the entry point –  Build the DU image from this Dockerfile

Solum abstrac-ons -‐ Deployment Unit (DU) (2/2)

•  Building a DU –  Start from the specified languagepack LP –  Specify execu-on of languagepack’s ‘build.sh’ –  Inject applica-on source code –  Use the run command specified in app defini-on as the default Entrypoint to

run the DU

•  LP and DU storage Glance, Swi[, Docker registry

18

Solum abstrac-ons -‐ Workflow

•  Abstrac-on to represent execu-on of applica-on deployment consis-ng of one or more workflow stages

•  Supported workflow stages

–  Build DU (and store it for future use) –  Run unit tests, build DU –  Run unit tests, build DU, deploy DU (if unit tests pass) –  Build DU, deploy DU –  Deploy a previously built DU (not yet available)

•  A workflow can be triggered from github webhooks

Solum abstrac-ons -‐ Add-‐ons

•  Services needed by an applica-on E.g.: rela-onal database such as Trove

•  DU parameters

Solum supports ability to pass service’s connec-on parameters to applica-on DU

Architecture and system details

High-‐level architecture

Solum API

Solum Worker

Solum Deployer Queue Queue

Queue

Solum Conductor

Heat

Solum Database

Glance, Swi[, Docker Registry

Builds LP and DU Deploys DU

-‐  Decoupled architecture

-‐  Asynchronous communica-on

Demo

h8ps://vimeo.com/143425822

Agenda



•  Part III –  Contribu,ng to OpenStack –  OpenStack CI system –  Development best prac,ces –  Hands-‐on exercise


Contribu-ng to OpenStack

Tools/Systems

•  Gerrit: Code review system –  h8ps://review.openstack.org/

•  Launchpad: Work (bugs/blueprints) tracking system –  h8ps://launchpad.net/

•  Github: Code repository –  h8ps://github.com/openstack

•  Git: Code contribu-on •  Mailing list: openstack-‐dev –  h8p://lists.openstack.org/cgi-‐bin/mailman/lis-nfo/openstack-‐dev

•  IRC: chat.freenode.net

Setup steps (1/2)

•  h8p://docs.openstack.org/infra/manual/developers.html

•  Create Launchpad account •  Join OpenStack founda-on as founda-on member •  Sign individual contributor’s license agreement •  Install git-‐review •  Install git •  Run git review -‐s

Setup steps: Problems and troubleshoo-ng

•  Problem: We don't know where your gerrit is. Please manually create a remote

•  Solu-on: – Create a username in gerrit if not already created – Manually set gerrit remote. E.g.: git remote add gerrit "ssh://[email protected]:29418/openstack/solum.git”

Setup steps: Problems and troubleshoo-ng

•  Problem: Problem running 'git remote update gerrit’

Fetching gerrit Permission denied (publickey). fatal: Could not read from remote repository. •  Solu-on: –  Generate sshkey and add it to gerrit –  ssh-‐keygen

Steps to submit a patch

•  Create a Launchpad bug or assign an exis-ng bug to yourself

•  Fixing the bug –  Itera-ve process

•  Understand the issue (be in touch with project developers on project’s IRC channel or on openstack-‐dev mailing list)

•  Design solu-on •  Write code •  Write tests •  Commit code •  Provide good commit message •  Submit patch for review

Steps to merge a patch

•  Patch needs to pass configured automated tests •  Patch needs to get two +2 votes •  Patch needs to be approved for merging by project’s core reviewer

OpenStack CI System

Con-nuous Integra-on of patches

•  Systems involved: – Zuul – Gerrit –  Jenkins – Github

•  Pre-‐merge checking – Patch is merged to master only if all the configured tests pass

OpenStack’s CI setup

Gerrit Zuul Jenkins

Github

2 3

4 5

1 Patch submi8ed for review

Patch picked up for tes-ng Tests ran on the patch

Jenkins gives its vote

Code merged

Patch review page details

Gerrit: Patch that fails gate tests

Development best prac-ces

Best prac-ces, tricks, -ps

•  Devstack •  Vagrant and Virtualbox •  Python •  Screen •  Debugging •  Git

Devstack

•  OpenStack environment consis-ng of all the openstack-‐services – Keystone, Nova, Heat, Solum, etc.

•  Can be run within a virtual machine or independently

•  Useful for development

Vagrant and VirtualBox

•  Vagrant –  Configura-on management tool for configuring virtual machines

•  VirtualBox –  Virtualiza-on system

•  Spinning up a development environment using Vagrant + VirtualBox –  h8ps://github.com/rackerlabs/vagrant-‐solum-‐dev –  git clone <solum-‐vagrant-‐repo> –  vagrant up -‐-‐provision

Sexng up Solum development env

•  Develop on Host, debug on the VM •  Install Vagrant and Virtualbox •  Clone solum and python-‐solumclient repositories on the Host

•  Set following environment variables: –  SOLUM=<path on Host system to solum repo> –  SOLUMCLIENT=<path on Host system to python-‐solumclient repo>

•  vagrant up -‐-‐provision •  vagrant ssh

Python

•  PyCharm IDE •  pdb – Put breakpoint in the code where you want to break •  import pdb; pdb.set_trace()

– pdb commands •  s: Execute the current line, stop at first possible occasion •  c: Con-nue execu-on, only stop when a breakpoint is encountered

•  h8ps://docs.python.org/2/library/pdb.html

Tes-ng

•  Run tests locally before submixng patch for review – From within solum directory •  tox –epep8 •  tox –epy27

•  Tox – Command line tool for running tests within virtualenv

Code paths on Devstack

•  solum /opt/stack/solum

•  python-‐solumclient /opt/stack/python-‐solumclient

•  Solum logs –  /var/log/solum/deployer –  /var/log/solum/worker

•  Devstack logs (of various screen sessions) –  /opt/stack/logs/

Screen

•  Program that allows sharing physical terminal between different processes

•  Entering Solum’s screen session in Devstack –  screen –x stack

•  Screen commands – Move to next window: Ctrl+a n – Move to previous window: Ctrl+a p –  See all windows: Ctrl+a “ –  Enter scrollback/copy mode: Ctrl+a [ –  Leave scrollback/copy mode: Ctrl+a ] –  h8p://www.pixelbeat.org/lkdb/screen.html

Debugging

•  Debug code/feature – Put breakpoints in the code –  Invoke code-‐under-‐test from CLI – Use screen to enter breakpoint and then debug using Pdb

•  Debug failing func-onal test – h8ps://review.openstack.org/#/c/254344/

•  Debug failing unit test

Git

•  Crea-ng dependent commits •  Squashing commits – git rebase -‐i HEAD~<number-‐of-‐commits> – pick/squash/edit

•  Upda-ng dependent commit and submixng for review

•  h8ps://ask.openstack.org/en/ques-on/31633/gerrit-‐best-‐way-‐to-‐make-‐a-‐series-‐of-‐dependent-‐commits/

Hands-‐on exercise

•  Submit first patch to OpenStack (45 minutes)

•  h8ps://etherpad.openstack.org/p/solum-‐low-‐hanging-‐fruit-‐bugs

Ques-ons

Contributing to OpenStack

Engineering

Transcript of Contributing to OpenStack