Build Release Deploy System Evolution in Juno

Learning to Scale OpenStack

Created by: Rainya Mosher

Modified Date: 05 NOV 2014

Classification: Public

Twitter: @rainyamosher | Freenode: rainya

2

In preparing for battle, I have always found that plans are useless, but planning is indispensable.

~ Dwight D. Eisenhower, 34th President of the United States

www.rackspace.com

3www.rackspace.comSpace Unicorn, 2012, Gripp & Drouhardhttp://youtu.be/17o1OlroNSE

4www.rackspace.comMiltonious Blog Unicorn of Technical Difficulities, Source Unknown

5www.rackspace.com

5 time zones6 production regions 6+ lower level environments

(CI, Test, PreProd)20,000+ hypervisors20,000+ computes2,000+ control plane nodes

6www.rackspace.com

User Request Nova, Glance, Neutron Services

XenServer, OVS

e.g., Build Instance

Use

r Acc

ess

to

Clo

ud I

nsta

nce Build Release

Deploy System

Upgrade OpenStack

ServicesCONTROL PLANE (CP)

DATA PLANE (CP)

OpenStack Magic

Customer VM

7

To date, the control plane has been upgraded 11 times in 2014.

www.rackspace.com

2014 Control Plane Upgrades

Iteration 92 deploysMigrate to Neutron

Iteration 106 deploysBoot from Volume

Iteration 11 3 deploys (planned)Expose Public Neutron API

DecOctSepAugJulJunAprMarFebJan May Nov

Iteration 83 deploysExpose Public Glance API

9www.rackspace.com

10www.rackspace.com

Upstream Nova, Glance, and Neutron project code

11www.rackspace.com

PLYGit-Based Patch Management

https://github.com/rconradharris/ply

12www.rackspace.com

Transitioning from Puppet to Ansible on a project-by-project basis

13www.rackspace.com

Upstream, launchpad.net

Internally, mixed use of:

• Redmine http://www.redmine.org

• Jirahttps://jira.atlassian.com

14www.rackspace.com

On Github Pull Request:

• runs unit tests

• validates issue # in commit

• updates issue tracker link

15www.rackspace.com

Internal GitHub provides

• secure place for sensitive settings

• sandbox to move product development forward

16www.rackspace.com

Package artifact

• Tarball of project code virtual environment + configs

• .tar artifact uploaded to package server

• distributed via torrent

17www.rackspace.com

Ansible playbooks triggered by Jenkins with python goodness

• Upgrade control plane

• Perform database changes

• Upgrade computes

18www.rackspace.com

19www.rackspace.com

20www.rackspace.com

Ope

nSta

ck

Con

tinuo

us

Inte

grat

ion

Mer

ge

Pat

ches

Pac

kagi

ng

Rac

kspa

ce

Con

tinuo

us

Inte

grat

ion

Dep

loy

to C

IA

utom

ated

Te

stin

g

Rac

kspa

ce

Rel

ease

B

ranc

h

Dep

loy

to

TES

TA

utom

ated

Te

stin

g

Rac

kspa

ce

Rel

ease

C

andi

date

Sch

edul

e

Mai

nten

ance

Dep

loy

to

PR

EP

RO

DA

utom

ated

Te

stin

gR

elea

se

Bra

nch

PR

Pac

kagi

ngR

e-D

eplo

y P

RE

PR

OD

Re-

Run

Te

stin

g

Rac

kspa

ce

Pro

duct

ion

Del

iver

y

Sch

edul

e

Mai

nten

ance

Dep

loy

to

PR

OD

Aut

omat

ed

Test

ing

DO

NE

Iteration CI/CD

Iteration Cycle

Iteration Change and Release Management

21www.rackspace.com

Ope

nSta

ck

Con

tinuo

us

Inte

grat

ion

Mer

ge

Pat

ches

Pac

kagi

ng

Rac

kspa

ce

Con

tinuo

us

Inte

grat

ion

Dep

loy

to C

IA

utom

ated

Te

stin

g

Rac

kspa

ce

Rel

ease

B

ranc

h

Dep

loy

to T

ES

TA

utom

ated

Te

stin

g

22www.rackspace.com

Rac

kspa

ce

Rel

ease

C

andi

date

Sch

edul

e

Mai

nten

ance

Dep

loy

to

PR

EP

RO

DA

utom

ated

Te

stin

gR

elea

se

Bra

nch

PR

Pac

kagi

ngR

e-D

eplo

y P

RE

PR

OD

Re-

Run

Te

stin

g

23www.rackspace.com

Rac

kspa

ce

Pro

duct

ion

Del

iver

y

Sch

edul

e

Mai

nten

ance

Dep

loy

to

PR

OD

Aut

omat

ed

Test

ing

DO

NE

24www.rackspace.com

Ope

nSta

ck

Con

tinuo

us

Inte

grat

ion

Mer

ge-P

ly

Pac

kagi

ng

Rac

kspa

ce

Con

tinuo

us

Inte

grat

ion

Dep

loy

to C

IA

utom

ated

Te

stin

g

Rac

kspa

ce

Rel

ease

B

ranc

h

Dep

loy

to

TES

TA

utom

ated

Te

stin

g

Rac

kspa

ce

Rel

ease

C

andi

date

Sch

edul

e

Mai

nten

ance

Dep

loy

to

PR

EP

RO

DA

utom

ated

Te

stin

gR

elea

se

Bra

nch

PR

Pac

kagi

ngR

e-D

eplo

y P

RE

PR

OD

Re-

Run

Te

stin

g

Rac

kspa

ce

Pro

duct

ion

Del

iver

y

Sch

edul

e

Mai

nten

ance

Dep

loy

to

PR

OD

Aut

omat

ed

Test

ing

DO

NE

• Not production scale

• Shared train leads to blocking issues

• Multiple trains = coordination overhead

• Production deploys stop the flow

25

The term "Microservice Architecture" has sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services [with] certain common characteristics […] around business capability, automated deployment, intelligence in the endpoints, and decentralized control of languages and data.

Martin Fowler

www.rackspace.com

http://martinfowler.com

26

•The Road to Minimally Impacting Live Upgrades of the Rackspace Public Cloud–TODAY from11:50 – 12:30– Room 252AB

•Building the RackStack: Packaging from Upstream OpenStack–TODAY from 15:30 – 16:10– Amphitheatre Bleu

Companion Sessions

www.rackspace.com

27

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

~ Principles Behind the Agile Manifesto

www.rackspace.com

http://www.agilemanifesto.org/principles.html

Twitter: @rainyamosher | Freenode: rainya

THANK YOU