Contributing to OpenStack

49
Tutorial on Contributing to OpenStack Dr. Devdatta Kulkarni PTL, OpenStack Solum [email protected]

Transcript of Contributing to OpenStack

Page 1: Contributing to OpenStack

Tutorial on Contributing to OpenStack

Dr. Devdatta KulkarniPTL, OpenStack Solum

[email protected]

Page 2: Contributing to OpenStack

Agenda

• Part I – Introductions– Introduction to OpenStack

• Part II – Deep dive of a OpenStack project - Solum

• Part III– Contributing to OpenStack– OpenStack CI system– Development best practices– Hands-on exercise

• Questions and Answers

Page 3: Contributing to OpenStack

Agenda

• Part I – Introductions– Introduction to OpenStack

• Part II – Deep dive of a OpenStack project - Solum

• Part III– Contributing to OpenStack– OpenStack CI system– Development best practices– Hands-on exercise

• Questions and Answers

Page 4: Contributing to OpenStack

Introduction

• Me– Dr. Devdatta Kulkarni– I go by Devdatta or Dev– Working at Rackspace USA– First contributor on Solum– Currently Project Team Lead (PTL) of Solum for Mitaka cycle– PhD, University of Minnesota Minneapolis

• You

Page 5: Contributing to OpenStack

Introduction to OpenStack

Page 6: Contributing to OpenStack

What is OpenStack?

A family of “open” projects that together enable different aspects of cloud computing

Openness of OpenStackhttps://wiki.openstack.org/wiki/Open

• Open source• Open design• Open development• Open community

Page 7: Contributing to OpenStack

OpenStack family

• Family of projects enabling different aspects of cloud computing– Authentication and authorization layer

• Keystone– Infrastructure-as-a-service layer

• Nova, Glance– Storage layer

• Swift, Cinder– Networking layer

• Neutron– Platform-as-a-service layer

• Solum, Heat, Murano, Mistral, Trove

• Project types– OpenStack big tent vs. OpenStack related projects

Page 8: Contributing to OpenStack

OpenStack by numbers

• 49 projects in OpenStack big tent• Several hundred organizations involved • Several thousand developers participating

Page 9: Contributing to OpenStack

Agenda

• Part I – Introductions– Introduction to OpenStack

• Part II – Deep dive of a OpenStack project - Solum

• Part III– Contributing to OpenStack– OpenStack CI system– Development best practices– Hands-on exercise

• Questions and Answers

Page 10: Contributing to OpenStack

What is Solum?

OpenStack project that provides easy mechanism for application developers to deploy and run their applications on OpenStack starting from application’s source code

Why should you care? (the value proposition)– For operators, Solum provides ability to make your OpenStack cloud more

useful for your clouds’ application developers by simplifying the process of deploying applications to it

– For developers, Solum provides an easy-to-use platform for building, testing, and deploying applications on OpenStack clouds

Page 11: Contributing to OpenStack

Project goals

Application Stack FlexibilityAbility to support applications written in different languages and using different application frameworks

Developer ProductivityAbility to perform CI/CD, integration with github

Add-On Services ExtensibilityAbility to support different add-on services

Application PortabilityAbility to deploy applications across different OpenStack clouds (use native OpenStack services)

Page 12: Contributing to OpenStack

Project details

• Wiki:https://wiki.openstack.org/wiki/Solum

• Meetings:Tuesdays 1700 UTC in #openstack-meeting-3

• IRC:Channel: #solum on freenode

• Code:https://github.com/openstack/solumhttps://github.com/openstack/python-solumclient

Page 13: Contributing to OpenStack

Solum abstractions

• App• Languagepack (LP)• Deployment Unit (DU)• Workflow• Add-on

Page 14: Contributing to OpenStack

Solum abstractions - App

Declarative application definition

• source code repository• run command• port• languagepack

version: 1name: cherrypydescription: python web applanguagepack: pythonsource: repository: https://github.com/rackspace-solum-samples/solum-python-sample-app.git revision: masterworkflow_config: run_cmd: python app.pyports:- 80

Page 15: Contributing to OpenStack

Solum abstractions - Languagepack (LP)

• Docker image which contains application specific build and runtime libraries

• A LP needs to implement Solum’s languagepack contractA well-known file available at a known location on the languagepack image

/solum/bin/build.sh

• How to build a LP?Dockerfile

• Operator-defined or user-defined

Page 16: Contributing to OpenStack

Solum abstractions - Languagepack Example: Python

FROM ubuntu:preciseMAINTAINER Murali Allada <[email protected]>

RUN apt-get -yqq updateRUN apt-get -yqq install python-pipRUN apt-get -yqq install python-dev

COPY build.sh /solum/bin/

https://github.com/rackspace-solum-samples/solum-languagepack-python

#!/bin/bash

# Check if pip is installedpip help[[ $? != 0 ]] && echo python-pip is not installed. && exit 1

# Install app dependenciescd /apppip install -r requirements.txt

build.shDockerfile

Page 17: Contributing to OpenStack

Solum abstractions - Deployment Unit (DU) (1/2)

• Docker image that is formed from the languagepack image with application’s source code added to it

DU = LP + application source code

• Solum guarantees the DU contractApplication source code available at a known location on the DU image /app

• How to build a DU?– Construct Dockerfile with languagepack as the base image, application source code

injected, run command as the entry point– Build the DU image from this Dockerfile

Page 18: Contributing to OpenStack

18

Solum abstractions - Deployment Unit (DU) (2/2)

• Building a DU– Start from the specified languagepack LP– Specify execution of languagepack’s ‘build.sh’– Inject application source code– Use the run command specified in app definition as the default Entrypoint to

run the DU

• LP and DU storageGlance, Swift, Docker registry

Page 19: Contributing to OpenStack

Solum abstractions - Workflow

• Abstraction to represent execution of application deployment consisting of one or more workflow stages

• Supported workflow stages– Build DU (and store it for future use)– Run unit tests, build DU– Run unit tests, build DU, deploy DU (if unit tests pass)– Build DU, deploy DU– Deploy a previously built DU (not yet available)

• A workflow can be triggered from github webhooks

Page 20: Contributing to OpenStack

Solum abstractions - Add-ons

• Services needed by an applicationE.g.: relational database such as Trove

• DU parametersSolum supports ability to pass service’s connection parameters

to application DU

Page 21: Contributing to OpenStack

Architecture and system details

Page 22: Contributing to OpenStack

High-level architecture

SolumAPI

SolumWorker

SolumDeployerQueue Queue

Queue

SolumConductor

Heat

Solum Database

Glance, Swift, Docker Registry

Builds LP and DU Deploys DU

- Decoupled architecture

- Asynchronous communication

Page 23: Contributing to OpenStack

Demo

https://vimeo.com/143425822

Page 24: Contributing to OpenStack

Agenda

• Part I – Introductions– Introduction to OpenStack

• Part II – Deep dive of a OpenStack project - Solum

• Part III– Contributing to OpenStack– OpenStack CI system– Development best practices– Hands-on exercise

• Questions and Answers

Page 25: Contributing to OpenStack

Contributing to OpenStack

Page 26: Contributing to OpenStack

Tools/Systems

• Gerrit: Code review system– https://review.openstack.org/

• Launchpad: Work (bugs/blueprints) tracking system– https://launchpad.net/

• Github: Code repository– https://github.com/openstack

• Git: Code contribution• Mailing list: openstack-dev

– http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev• IRC: chat.freenode.net

Page 27: Contributing to OpenStack

Setup steps (1/2)

• http://docs.openstack.org/infra/manual/developers.html• Create Launchpad account• Join OpenStack foundation as foundation member• Sign individual contributor’s license agreement• Install git-review• Install git• Run git review -s

Page 28: Contributing to OpenStack

Setup steps: Problems and troubleshooting

• Problem:We don't know where your gerrit is. Please manually create a remote

• Solution:– Create a username in gerrit if not already created– Manually set gerrit remote. E.g.: git remote add gerrit "ssh://[email protected]:29418/openstack/solum.git”

Page 29: Contributing to OpenStack

Setup steps: Problems and troubleshooting

• Problem: Problem running 'git remote update gerrit’

Fetching gerrit Permission denied (publickey). fatal: Could not read from remote repository.• Solution:– Generate sshkey and add it to gerrit– ssh-keygen

Page 30: Contributing to OpenStack

Steps to submit a patch

• Create a Launchpad bug or assign an existing bug to yourself• Fixing the bug– Iterative process

• Understand the issue (be in touch with project developers on project’s IRC channel or on openstack-dev mailing list)

• Design solution• Write code• Write tests• Commit code• Provide good commit message• Submit patch for review

Page 31: Contributing to OpenStack

Steps to merge a patch

• Patch needs to pass configured automated tests• Patch needs to get two +2 votes• Patch needs to be approved for merging by project’s

core reviewer

Page 32: Contributing to OpenStack

OpenStack CI System

Page 33: Contributing to OpenStack

Continuous Integration of patches

• Systems involved:– Zuul– Gerrit– Jenkins– Github

• Pre-merge checking– Patch is merged to master only if all the configured tests

pass

Page 34: Contributing to OpenStack

OpenStack’s CI setup

Gerrit Zuul Jenkins

Github

2 3

45

1Patchsubmittedfor review

Patch picked upfor testing Tests ran on the patch

Jenkins gives its vote

Codemerged

Page 35: Contributing to OpenStack

Patch review page details

Page 36: Contributing to OpenStack

Gerrit: Patch that fails gate tests

Page 37: Contributing to OpenStack

Development best practices

Page 38: Contributing to OpenStack

Best practices, tricks, tips

• Devstack• Vagrant and Virtualbox• Python • Screen• Debugging• Git

Page 39: Contributing to OpenStack

Devstack

• OpenStack environment consisting of all the openstack-services– Keystone, Nova, Heat, Solum, etc.

• Can be run within a virtual machine or independently

• Useful for development

Page 40: Contributing to OpenStack

Vagrant and VirtualBox

• Vagrant– Configuration management tool for configuring virtual machines

• VirtualBox– Virtualization system

• Spinning up a development environment using Vagrant + VirtualBox– https://github.com/rackerlabs/vagrant-solum-dev– git clone <solum-vagrant-repo>– vagrant up --provision

Page 41: Contributing to OpenStack

Setting up Solum development env

• Develop on Host, debug on the VM• Install Vagrant and Virtualbox• Clone solum and python-solumclient repositories on the

Host• Set following environment variables:– SOLUM=<path on Host system to solum repo>– SOLUMCLIENT=<path on Host system to python-solumclient repo>

• vagrant up --provision• vagrant ssh

Page 42: Contributing to OpenStack

Python

• PyCharm IDE• pdb– Put breakpoint in the code where you want to break• import pdb; pdb.set_trace()

– pdb commands• s: Execute the current line, stop at first possible occasion• c: Continue execution, only stop when a breakpoint is

encountered• https://docs.python.org/2/library/pdb.html

Page 43: Contributing to OpenStack

Testing

• Run tests locally before submitting patch for review– From within solum directory• tox –epep8• tox –epy27

• Tox– Command line tool for running tests within virtualenv

Page 44: Contributing to OpenStack

Code paths on Devstack

• solum/opt/stack/solum

• python-solumclient/opt/stack/python-solumclient

• Solum logs– /var/log/solum/deployer– /var/log/solum/worker

• Devstack logs (of various screen sessions)– /opt/stack/logs/

Page 45: Contributing to OpenStack

Screen

• Program that allows sharing physical terminal between different processes

• Entering Solum’s screen session in Devstack– screen –x stack

• Screen commands– Move to next window: Ctrl+a n– Move to previous window: Ctrl+a p– See all windows: Ctrl+a “– Enter scrollback/copy mode: Ctrl+a [– Leave scrollback/copy mode: Ctrl+a ]– http://www.pixelbeat.org/lkdb/screen.html

Page 46: Contributing to OpenStack

Debugging

• Debug code/feature– Put breakpoints in the code– Invoke code-under-test from CLI– Use screen to enter breakpoint and then debug using

Pdb• Debug failing functional test– https://review.openstack.org/#/c/254344/

• Debug failing unit test

Page 47: Contributing to OpenStack

Git

• Creating dependent commits• Squashing commits– git rebase -i HEAD~<number-of-commits>– pick/squash/edit

• Updating dependent commit and submitting for review

• https://ask.openstack.org/en/question/31633/gerrit-best-way-to-make-a-series-of-dependent-commits/

Page 48: Contributing to OpenStack

Hands-on exercise

• Submit first patch to OpenStack (45 minutes)

• https://etherpad.openstack.org/p/solum-low-hanging-fruit-bugs

Page 49: Contributing to OpenStack

Questions